Anonymous Methods in C# 2.0: What’s Old is New Again!

Paul Kimmel recently wrote an article for informit.com entitled "Anonymous Methods in C# 2.0" where he made the following statement:

Linguistically, I love methods, but as a practical matter anonymous methods may
just be an example
of some inventive person at Microsoft being a bit too clever.

Unfortunately that was the pull-quote used to promote the article on another part of the site. Unfortunate, because I think they are one of the best new features in C#.

I won’t take Paul to task here - I’ll just assume he’s never worked with a similar language feature in any other language, but I was extremely excited when I first saw they were adding anonymous methods to C# 2.0. Anonymous methods are similar to the Smalltalk concept of CodeBlocks which I learned of and fell in love with during my Clipper 5.0 programming days in the early 90’s. Clipper 5.0’s Codeblocks were my favorite language feature because they provided so much power with so much simplicity.  Even that crazy new language Ruby has (Code)Blocks!

I remember the reason they added CodeBlocks to Clipper was to support "User Defined Commands (UDCs)."  UDCs allowed the developer to use the old dBase-style commands and have them translated via a preproccessor into function calls with parameters (I always love it when language designers actually use a language! When they do, they add really useful features!)

So, looking at the dBase REPLACE command, which is roughly equivalent to a SQL UPDATE, the UDC would convert this:

REPLACE Name WITH "Mike Schinkel" FOR Status="X"

Into something like this (forgive me if it’s not 100% correct, it’s been a looong time since I coded in Clipper):

dbreplace( {|| Name:= "Mike Schinkel" }, {|| Status="X" } )

Try doing that without a language feature like CodeBlocks!

During my Clipper 5.0 days, I found the basic benefits of a (Code)block/Anonymous Function are three-fold:

  • They eliminate the need to uniquely name a method when it would probably be called only once
  • They reduce the number of method names you need to track,
  • They allow you to define code physically closer to the using code. 

Maybe I’m wrong, but I’ll bet Paul will soon find anonymous methods to be one of the most useful language features added to C# in v2.0; right up there with Generics!

Optimal Dressing, or A challenge to Eric Lippert

Ah the memories

Eric Lippert’s blog post about partial order sorting was, as always, interesting and well written. However, this specific post brought back memories of a project I did for Gateway Foods of Lacrosse Wisconsin back in the early 90’s. (I googled them and they don’t appear to no longer exist.)

Though my project was not as fun or easy to understand as Eric’s example, Gateway wanted to perform the "hot and new" business analysis called Activity-Based Costing, or ABC. Instead of just looking at which food items were most profitable, they looked at which customers were most profitable.

Aced-out by the SQL dude

They hired one of the big eight firms (remember when they were called "the big eight?) Booze-Allen-Hamilton to do the analysis and then went looking for a programmer. I heard about the project from my contact "George" for whom I had delivered a training course on Clipper earlier. I got excited, but really had no chance at it because the IT department didn’t believe in PCs, so they hired a SQL programmer to implement.

A few months went by and George called to tell me the project was going no where. A few more later George cakked to tell me the project was in real trouble. A few month more still, and I got a call from George asking if I could I start on the project that week in hopes of salvaging it (of course they had already spent most of the budget on the SQL guy, but what can you do?)

Be careful what you wish for

Later that week I arrived and met with the people from Booze, and they proceeded to hand me a stack of paper 10 inches thick that contained the allocation formulas! (I kid you not.) My job was to write a Clipper program to apply those formulas to a HUGE 300Mb database (remember, this was the early 90s and 300Mb was huge back then.) The allocation formulas were for things like "Allocate the cost of fuel to deliver to each customer based on the miles required to travel to customer. Sum the cost of fuel and multiply by the miles traveled for each customer divided by total miles traveled to all customers." And so on. Around 1800+ formulas in all.

For the next three days, my eyes glazed over as they tried to understand the formulas. I starting thinking "What the hell have I got myself into? There is no way I will ever be able to understand all of these formulas well enough to encode them into a program. No wonder the SQL guy failed. I am screwed too!" But finally, it started to dawn on me; "Don’t learn the formulas, learn the patterns." I’ve always been extremely good at pattern recognition so why didn’t I see it immediately? (which contrasts nicely with all the other things at which I’m extremely bad.)

Patterns; what a concept!

The initial database had number tables like fooditems, customers, and order. It turned out there were only three patterns:

  1. Scan a table and perform a calculation like sum or average and write the output to a one-record table. Examples included calculating total sales, average order size, etc.
     
  2. Scan a table and perform a calculation like sum or average for each fooditem, or customer, or similar, and write the output to a table that has one record for every fooditem, customer, or similar.
     
  3. Scan a table and perform a calculation like every record and write the output to a table with the same number of records as the scanned table.

That’s it. Three patterns. All 1800+ formulas boiled down to three patterns. SQL gurus know the following are examples of the above three patterns:

  1.    SELECT
            SUM(SubTotal) AS TotalPrice
       INTO
            OneRec
       FROM
            Orders
  2.    SELECT
            CustomerID,
            SUM(SubTotal) AS TotalOrdered
       INTO
            OnePerCustomer
       FROM
            Orders
       GROUP BY
            CustomerID
  3.    SELECT
            opc.CustomerID,
            opc.TotalOrdered/c.TotalDeliveryMiles AS SalesPerMile
       INTO
            CustomerFactors
       FROM
            Customer c INNER JOIN
            OnePerCustomer opc ON opc.CustomerID=c.ID

Technique triumphs over toolset

Ultimately I succeeded using Clipper where the SQL programmer had failed. He didn’t fail because he used SQL and my success was not because I used Clipper. He failed because he tried to hand-code all 1800+ formulas and I succeeded because I wrote a Clipper program to handle the three patterns, not the 1800+ formulas. In hindsight given the tools available at the time, the best solution would probably have used Turbo Pascal and SQL, but such is life.

The Gateway people entered the allocation formulas from Booze into a grid in my program, my program then determined which formulas to calculate and in which order, and then program executed those formulas. I used Clipper’s "macro" capability to allow me to execute the text formulas much like how Execute() can be used in VBScript. However, VB.NET programmer could do the same thing today by writing a program that generates SQL and then passing that SQL to ADO.NET to execute.

Back to Eric’s Post

The "which formulas in which order" was the part I was reminded of by Eric’s post. It actually took me three weeks to write that first program. 12 hours a day. 7 days a week. In the dead of winter. In Lacrosse Wisconsin. Brrrrr. (I’m from the south in Atlanta and I am a wuss when it comes to frigid weather!) Today I could probably write that same program in VB.NET and SQL in an afternoon. But then hardware is a lot faster now.

Intermediate Representations

Why three weeks? My program took almost two days to apply the 1800+ formulas to the 300Mb database! Ever try to debug a program where you have to step through code for 24 hours before you get to see the next bug? Not very productive. It was here I actually learned the tremendous value of creating an "intermediate representation." HTML is a perfect example: one program can generate HTML and another can display it. SQL, XML, and MSIL are also immediate representations. One of the benefits of an intermediate representation is you can validate output without having to observe behavior. In essense intermediate representations provide much the same benefits as state machines.

I split my program into two modules; one that compiled the formulas, and a second that executed the compiler’s output. My "compiler" took only about 2 hours to run which allowed me to debug my program in this lifetime. It still took two days to execute the formulas, but at least those formulas executed correctly!

Just when I thought I was out…

I went home victorious. Or so I thought. A few weeks later I get a call. (They tracked me down while I was on vacation, no less!) The Gateway people decided they wanted to modify the formulas and run "what-ifs." After each "what-if" it took them two days to get an answer. "Couldn’t I make this thing any faster?" (Imagine if the SQL guy had succeeded in encoding all those formulas into a SQL script? He would have had a job for life! A boring-as-hell job, but job nonetheless. Until Gateway realized all the money they were wasting and then threw him out on his ear.)

My original program, except for the compiler part, was almost exactly what Eric Lippert blogged about a few days ago. I was damn proud I wrote it. However, optimizing it was much, much harder. Turns out what was happening was each pass on each of the tables took a really, really long time (we knew that), but also there were formulas that could be performed on the same pass, but I doing a complete pass for each formula.

Well, to finally cut a long story short, another three weeks and I was able to implement an optimizer which batched together formulas that could run at the same. That version took only about 6 hours to run. Good enough, and that was the last of Gateway foods for me (I never was very good at milking clients; guess that’s why never made much money while in consulting.)

Optimal Dressing

So, back to Eric’s example. Let’s assume it took one minute for each seperate Eric put on, but if he was able to put on multiple items "at the same time", then each batch of items would only takes him 1.5 minutes to put on. His original program’s output, which dressed him in this manner:

      tophat      : [ ]
      shirt       : [ ]
      bowtie      : ["ashirt"a]
      socks       : [ ]
      vest        : ["ashirt"a]
      pocketwatch : ["avest"a]
      underpants  : [ ]
      trousers    : ["aunderpants"a]
      shoes       : ["atrousers"a, "asocks"a]
      cufflinks   : ["ashirt"a]
      gloves      : [ ]
      tailcoat    : ["avest"a]

This would take Eric 12 minutes to dress. However, if he batched items that could be done at the same time (i.e. items that had no prior dependencies), Eric could dress in 4.5 minutes:

      tophat      : [ ]
      socks       : [ ]
      shirt       : [ ]
      underpants  : [ ]
      gloves      : [ ]

  then

      trousers    : ["aunderpants"a]
      bowtie      : ["ashirt"a]
      vest        : ["ashirt"a]
      cufflinks   : ["ashirt"a]

  then

      pocketwatch : ["avest"a]
      shoes       : ["atrousers"a, "asocks"a]
      tailcoat    : ["avest"a]

The Challenge:

So here’s my challenge Eric (and I hope you don’t hate me for it): Implement an optimizer that will generate batched output where items in each batch have no interrelated dependencies, as shown in the last example above. It was a bitch to write in Clipper; how about in JScript?

A little background is in order…

As my second post I guess I’ll tell potential readers a little about myself and my company.

I live in Atlanta, Georgia USA where I have lived most of my life. From 1981 to 1988 I attended Georgia Tech to get a degree in Mechanical Engineering, and an "honorary" degree in Computer Science. I gave myself that "honorary" degree because I found out too late that I loved computers and hated engineering. I took many ICS[1] classes and aced all of them except in assembler (I choked on the final and got a "B"), but none of my ICS classes counted towards my ME degree. My grades in my ME classes were, shall we say, less than stellar; I had a mantra as my academic career started coming to a close: "D" for "D"egree!  If only I had switched my major early on…

From 1987 to 1990, I together with two others formed a partnership called The DSW Group for consulting and training Nantucket’s Clipper, now a product of Computer Associates[2].

Clipper was/(is) a DOS-based programming lanaguage that evolved from its humble beginnings to become quite an interesting language. It’s main competitor was FoxPro, and I cried when CA bought Nantucket, and Microsoft bought Fox instead of Clipper.

While at DSW I focused on having DSW become the leading training and consulting company in Clipper. In those days, I was very much a head’s down programmer and very pendantic about "my" Clipper language and the “right“ way to program. Let’s just say a mere mention of FoxPro by a visitor almost got him chased out of the office!

Around 1990 I developed wanderlust and, leaving The DSW Group in the worthy hands of my former partners Terry Dietzler and Ed Weber, went to the Washington DC area.  In northern Virginia, ironically my birthplace, I went to work as Director of Training for Financial Dynamics (FDI) with Michael Horwith and Steven Delbianco (now VP for Public Policy at ACT.)  FDI was acquired by Spectrum Technology Group in November 1997.

While in the DC area I met John Kaster (John is now with Borland Developer Relations) and he agreed to put up with my perfectionism and help me finish a book[3] on Clipper to be published by Addison Wesley. After what appeared to be eons Programming in Clipper 5 was born, which achieved critical acclaim but alas reached the market far too late to be a commercial success.

After a year in DC, I guess I became homesick and decided to head back to Atlanta. Though I loved working with Michael and Steve at Financial Dynamics, I felt the need to start a gig exclusively my own, which I named Expert Education (EEI).

I ran EEI as a Clipper training company until the end of 1994. By that time nobody wanted to be trained on Clipper anymore. This was sad because Clipper had some awesome language features, some of which are just starting to show up in .NET, and the Windows version which they called Visual Objects was even far cooler, but alas it was being planned and marketed by Computer Associates, not Microsoft. You get the picture.

However, in late 1993, having been frustrated with EEI’s inability to market our training classes w/o the use of CA’s mailing list (sometimes they would let us use it, other times they would not), I recognized something special in the Clipper-Only Tools catalog of 3rd party tools produced by a company called Zac[4]. Within six months they practically owned the ability to market to Clipper developers. Knowing I could not compete with them in Clipper, I (thankfully) decided to target a market for 3rd party tools for a product I knew absolutely nothing about how to program: Visual Basic!

So in March 1994 I formed a seperate company and named it VBxtras to produce a printed catalog of tools for Visual Basic developers. We christened it “The Ultimate Tools Catalog for Visual Basic.“  And it was.  One of the main reasons to start VBxtras was to promote Expert Education’s new Visual Basic training classes. Of course after little more than six months I decided to heck with training and to pour everything into VBxtras.

I took off my programmer hat and put on my marketer hat. We launched the first catalog in June/July 1994, and accelerated from there. How did it do? Well, I was lucky enough to put together a great team and, let’s just say, we took a wild ride all the way by 1999 to be listed as #123 on the Inc 500 list. Of course we changed the name to Xtras, Inc. in 1995 in order to dabble in other things besides Visual Basic, but none of them went anywhere so in 1999 we decided to focus back exclusively on Visual Basic developer tools.

In 1994 we offered tremendous value to both developer and vendor. We made a market where one had not previously existed as resellers of the day were paying no attention to “that toy language” Visual Basic. With VBxtras, I was a developer who liked "toys" (the components, not the language) and I wanted a reference guide that helped me quickly find and select from every single one available. In those days I didn’t worry much about profit, I just worried about getting every single vendor’s products into our catalog, and about providing as much information as possible that a developer would want so he could choose to which one met his needs the best.

That is until Microsoft decided to start promoting .NET. As I learned about .NET, my desire returned to program full time (though I doubt I ever will (be able to.)  .NET had most of the cool things that Clipper and Visual Objects had, but it came from Microsoft, and was (at least partially) designed by Anders Heilsberg, the man behind both Delphi and Turbo Pascal[5].

But .NET concerned us as we feared it could badly affect our business model. After all, "C# programmers are never going to buy tools from a placed called ‘VBxtras.’" So we decided to launch a new product line complete with a new name: Xtras.Net. We dubbed Xtras.Net "Your Resource for Quality .Net Tools" and launched a huge printed catalog of 3rd party tools for .NET in 2002.  So how has Xtras.Net done thus far? Well, frankly, not as well by comparison as did VBxtras during its first few years.

With Xtras.Net we had 8+ years of experience, but also 8+ years of baggage. When we started VBxtras, we had nothing else to distract us.  With Xtras.Net, running VBxtras distracted us.  Plus, when we started VBxtras the whole company had that "new start-up smell!" The staff when we started Xtras.Net had been here a while and was no longer a start-up staff.  It wasn’t possible to get 5+ year employees to put in all 16 hour days for months on end so that we could achieve the same goals we had for VBxtras in the beginning: to be The #1 Reference Guide for .NET.

So we haven’t done as well as I would have liked. Plus the world has changed greatly. In 1994 developers could not go from Google straight to a vendor’s website, pay for software on the spot , and download the bits and an unlock key. But today they can[6]. And in 1994, we didn’t have to compete with a venture capital backed dot-com that probably lost a dollar on ever dollar in sales for its first five years in business, all while we had to break even each month[7].

But we also haven’t done badly either, it is just a new world. I’m frustrated because we could have made a much greater positive impact for .NET developers than we have thus far, but by other’s accounts, we’ve done quite well.  We, like everyone else whose business has been negatively impacted by the Internet, have had to evolve, and will continue to evolve a lot more in the coming years.

As a matter of fact, some of the things we have planned for our evolution and that we are finally close to implementing have me tremendously excited. They excite me because I believe they can allow Xtras to transcend its current business model, and more importantly, impact developers and development in a very positive manner over the next several years. How do I know these things would be positive? Because I’m a developer at heart, and the things we have planned are things the developer in me would desperately love to see come to pass.

So thanks for reading, and stay tuned. Xtras next 10 years should be an even wilder ride than the last. In a good way, that is. :-)

 Footnotes

  1. ICS: Information and Computer Science
     
  2. Computer Associates: Where old software goes to die.
     
  3. Programming in Clipper 5: Of course Amazon has continued to this date to list the authors as me, Ed Weber, and Terry Dietzler even though the latter two dropped out at the beginning and John, whose name is on the cover, is the real co-author. I’ve even contacted Amazon about this, but to no avail.
     
  4. Zac Software: Since acquired by Global Computer, and about a year later, unceremoniously closed.
     
  5. I cut my programming teeth on Turbo Pascal in college.
     
  6. Google: Developers that go straight to vendors and by pass resellers pay full price, and don’t gain any of the benefits resellers offer. One of those benefits is resellers will act as their advocate when they have a problem with the vendor, among other things which I’ll cover over the coming months in future posts. I know this may sound self-serving, but hey, it’s my blog! Seriously though, if you’ll give me the benefit of the doubt and keep up with the subject over the next several months, I might be able to cast a different light on it for you. And I promise you, it won’t be my only topic (not even close!) as what I really want to talk about is programming in .NET and related.
     
  7. VC-backed dot-com: Yeah, but now they’ve run out of all that outlandish VC and they actually have to make profit! Ha! Let the games begin!