Clarifying my Microsoft Developer Division Rant

Contents


I Ranted and Eric Rebutted

The day before yesterday I wrote a long winded and rambling rant about how Microsoft’s release cycle and process for creating developer tools. I commented on how I believe it is making them fall behind and causing many formerly loyal Microsoft developers to look at open source solutions on non-Windows platforms. I referenced a post that Microsoft’s Eric Lippert wrote over 2.5 years ago titled Top Minds Are Working On It[1]. In retrospect it might have appeared I was being critical of Eric but that wasn’t my intention, and if that’s how I came across I apologize. Instead I was referencing Eric’s comments as symptomatic of the Microsoft culture at large. And yesterday I awoke to find that Eric had issued a rebuttal.

While I Respect the People at Microsoft…

But before I address his comments let me talk about Eric and all the others I’ve met from Microsoft. Eric is actually a brilliant guy, and very likeable. I’ve met Eric face-to-face and it’s obvious he’s much smarter than me. But then I could say that about most of those I’ve met from Microsoft; they don’t hire dummies. As a rule I’ve been impressed with every Microsoft employee I’ve met. They are super bright and AFAICT they do really want to do "Good Things(tm)".

…They Become Detached

But group dynamics being what they are, when you get a group of super-bright people together they become competitive, hone their debate skills, and learn to be strong advocates for whatever their own positions. And they can become detached from the outside world, much like politicians in high office. Politicians are also typically super bright, and most enter office wanting to do Good Things(tm), but once "on the inside" they loose touch with the concerns of their constituents. So I am not condemning Eric and his Microsoft colleagues, I am merely commenting on the culture that results and collectively channels them.

Eric Unconsciously Supports my Thesis!

So I started yesterday’s post saying I’d been planning to blog on the topic for a while but the reality was I still didn’t know how best to explain my concerns. But for better or for worse I did write a rambling essay yesterday, but ironically Eric’s comments made my points far better than I! My central thesis was that Microsoft isn’t meeting developer’s needs because of their processes and infrequent releases and consequently open-source alternatives are meeting developer’s needs instead. Eric’s both debated my examples and pointed out they now plan to address some issues I referenced. But not only did Eric not address my central thesis, he ironically supported it given his rebuttal’s choice of focus! In my post I wrote the following:

"Microsoft’s culture is to argue semantics when reality doesn’t match their world view"

And Eric’s comments proceeded to do exactly that! In my post from 2.5 years ago[1] I called for Microsoft to address things that PHP, Ruby, and Python are addressing today, which Eric rebutted at the time. In yesterday’s rebuttal Eric referenced his earlier comments stating (emphasis mine):

"Second, my ‘esoteric’ reasons for not implementing a scripty version of VB on the .NET platform were hardly esoteric then and are hardly esoteric now. They are (1) fracturing the language further causes confusion amongst customers and is massively expensive for little gain, …"

Dismissing the Proposal, Not Solving the Problem

Now I’ll freely admit Eric is far more qualified to evaluate my suggestion on technical merit, but that wasn’t the point of my 2.5 year old post. Customers with needs a company’s not addressing will often propose solutions they believe will address their needs, yet often their suggestions aren’t workable for whatever reason. People who specialize in addressing customer needs know that rather than dismiss suggestions as unworkable it’s far better to determine the customer’s actual needs and implement a workable solution instead. And often, many other customers have those same needs. So Eric dismissed my proposed solution but didn’t address my unresolved needs that prompted the proposal. I don’t attribute this failing to Eric, I attribute it to Microsoft’s current culture.

Not More Power; Transitionality!

Eric then went on to say:

"(2) we can do things to make VB.NET more powerful without fracturing the language,…"

Ironically, I didn’t ask for a more powerful VB.NET; I asked for one that was easier to start using and one that developers could then easily transition to more powerful usage. Though they believed they were providing an easier to use Visual Basic 2005, they addressed the language but not how people develop applications. Though they made strides with the Express Edition, my biggest concern was with the complexity of the development environment and the language. I suggested an interpretive environment with a transitional language design that allowed new developers to start easily yet be able to effortlessly grow their expertise with use. What I envisioned was something like Boo, but I wanted it 2.5 years ago with a simple interpretive environment, and I wanted it from Microsoft so that it could possibly generate a large and immediate and user base with a thriving community and significant peer support.

Today’s Potential Didn’t Address Yesterday’s Deficiency

Eric continued with the following…:

"(3) Microsoft makes scripting languages like Iron Python"

…but omitted the fact that a robust Iron Python was just a gleam in Jim Hugunin’s eye 2.5 years ago and is still not ready for prime time. Further, Microsoft’s approach is to host IronPython in Visual Studio which does nothing to bypass the complexity of Visual Studio!

Nor Does an Orphan Address Yesterday’s Deficiency

Eric then said:

"…and JScript .NET, use the right tool for the right job."

To which I did a double take wondering if he were really serious! JScript .NET is such the orphan that I can’t even believe he suggested it! The JScript .NET newsgroup has less than a screen full of messages, JScript .NET hasn’t been updated since 2003, and nobody’s even written about JScript .NET in years! So could Eric really have been serious when he suggested JScript .NET? Well, assuming he was, then:

  • JScript .NET does NOT have an easy-to-use interpretive environment, and
  • JScript .NET is a complex language; NOT simple-to-use, and
  • Again, JScript .NET has NO user-base!

And a Potential isn’t a Solution

Moving on Eric said:

"The .NET framework is already amenable to the development of scripting languages."

So why do we still not have a viable scripting solution for .NET supported from Microsoft more than half a decade after the .NET Framework’s first release?

Yes, there’s Powershell, but…

Okay, that’s not quite true. Eric didn’t mention it but in the spirit of honest debate, there is Microsoft’s PowerShell. But while I will freely admit PowerShell is really nice, PowerShell:

  1. Was only just released so doesn’t address the past 2.5 years,
  2. Doesn’t have a development environment,
  3. Can’t be used for web development,
  4. Doesn’t have a compiler for creating components for use in other .NET languages.
  5. Doesn’t have transitionality allowing it to scale up for much more complex projects as the developer’s experience grows.

A Correct yet Irrelevant Point

Eric then makes a point about "scripting languages" vs. "dynamic languages" (emphasis mine):

"Third, I want to make a distinction between scripting languages (languages intended to script things) and dynamic languages (languages which admit a type system which cannot be deeply analyzed at compile time.) Scripting languages are often dynamic languages, however it is entirely possible to use dynamic languages for tasks other than scripting. "

Okay… So Eric’s points are very technically valid, but they are totally irrelevant! Frankly I wasn’t asking for a language that was "intended to script things," I was proposing a language (and IDE) that would be:

  1. Productive,
  2. Easy to start using, and
  3. Scalable as one’s skills evolve.

Call it "scripting", call it a "dynamic language", call it whatever; it’s irrelevant. What is relevant is for it to be productive, easy, and scalable. Microsoft could choose to get there however they will, bit like arguing the semantics of "Car" vs. "SUV" with someone who just needs transportation, Eric’s distinctions were simply irrelevant to the needs. Totally unrelated, I ran across this joke yesterday. What could be more ironic?

Interest Doesn’t Necessarily Change Process

Eric finishes his prior point with:

"The VS team is VERY interested in understanding how to make the platform more amenable to dynamic languages."

Great! But are they going to actually engage people who are not .NET developers in the design of said dynamic languages and their respective development environments and then incorporate their feedback? Or is the VS Team just going to plug another dynamic language into Visual Studio? If the latter they will do so ignoring that Visual Studio users already have the language(s) they need and that there are at least an order of magnitude more people for whom Visual Studio is too overwhelming.

A Solution Offered; Wrong Product, Years from Now

A couple other comments Eric made were:

"C# 3.0 will have the "one line auto-implemented properties" feature you requested for VB. Enough people asked for it, we put it in. I do not know if VB will be doing the same. You’re welcome. "

and

"Current C# 3.0 features move in the direction of dynamic languages without actually making the language dynamic (lambdas, improved generic method type inference, extension methods, implicitly typed locals, anonymous tuple types). All of these however are implemented so as to keep the language statically analyzable. We are considering features for C# 4.0 which would make the language more dynamic without losing that important statically analyzable core."

That’s well and good, but it doesn’t address VB.NET, and it also makes my own point that Microsoft’s release cycles are too far apart! There are badly needed enhancements and they need to get them to developers more often than once every three years. C# 3.0 is still a way’s out, people need something today, and I personally wonder if I’ll even care about programming by the time C# 4.0 is released. Hell, I hope to have made my fortune and be retired by then! :-) But seriously, C# is a professional developer’s language and adding features to a professional developer’s language wasn’t even close to what I was proposing.

Too Little, Too Late: Acknowledged

In the last paragraph, Eric finally gives tacit acknowledgement of the concerns I raised in yesterdays post:

"Now, maybe these features aren’t what you want, or are too little too late, or the release schedule is too long for your liking, or whatever. That’s unfortunate."

Exactly. The release cycle needs to be compressed by an order of magnitude as it has been at the competition. More on this in a bit.

Studying Users Isn’t Feeling their Pain

Eric then signs off with:

"However I take exception to the claim that we do not study what real users are doing and try to provide tools that do what they want. We do that every day."

With this Eric either misunderstood my point, or more likely I didn’t state my point in a manner that was understandable. Whichever the case, let me clarify. I know the VS Team works hard to study real user’s needs every day. But what I also know is that the people who make the decisions about what gets released and when have considerations that are far different from the needs of developers. And it is human nature for one to place solving one’s own pains ahead of solving the pains of others as people simple can’t fully comprehend pains they don’t experience.

Trade-offs that Shortchange Developers

Eric let’s be concrete; if solving a customer problem today will cause the VS team a problem tomorrow they won’t do it. For example if the VS Team plans to rearchitect something next version they won’t provide an interface that developers need in this version if doing so will make that rearchitecture difficult. Some would say this is good product engineering to which I would actually agree, but it is nonetheless a trade-off that keeps developers from getting what they need today.

Experiencing Pain Empowers Real Solutions

So, if you’ll reread my post from yesterday you’ll see that I didn’t say "Microsoft doesn’t listen to customers"; I know damn well they do. On the contrary, I said that Microsoft’s Developer Division "Don’t solve real world problems." And the reason you don’t is because you don’t experience the pain that those real world problems cause. By comparison most developers contributing to open-source projects are doing so to solve pains which they themselves have, and that is why they are addressing developer needs much faster than Microsoft’s Developer Division.

Yes it is a Paradox…

An observant person would say that I’ve present a rather intractible dichotomy for Microsoft’s Developer Division; i.e. they need to use the developer tools they build to solve real world problems yet they also need to to develop those tools ten times faster! Now I’m sure these comment will cause the blood of those in Microsoft’s Developer Division to boil thinking I believe it’s possible they do their work ten times faster and do real world projects. But I was not advocating that; I’m fully aware of the myth of the "man-month" in software development projects.

…But Solve it You Must, Or Else

What I was pointing out, however, is if Microsoft’s Developer Division maintains its status quo they will slip farther behind and the loss of developers to other platforms will accelerate. And since developers, developers, developers have always been Microsoft’s life blood it is critical they address this issue. Every developer they loose to Linux or the Mac doubly weakens Windows. Microsoft must address this issue if they are to maintain the same level of relevancy during the next twenty years that they’ve had for the past twenty years. And most of what I’m suggesting isn’t complex, and it isn’t new. They probably have most of what they need already written and used internally. They just need to rethink what they are offering.

Change, or Be Changed

So to wrap up this second long-winded essay:

Microsoft’s Developer Division needs to implement drastic changes sooner than later. If they do not, outside forces will soon impose drastic changes upon them and it’s certain they will find those imposed changes to be far more painful. Or as the mechanic on the old Fram Oil Filter commercial used to say "Pay me now, or pay me later."

What if I’m Not Wrong?

P.S. If you’re from Microsoft’s Developer Division and choose to dismiss my concerns, just ask yourself this: What’s the downside for you if I’m right?

  1. Eric Lippert’s post was in response to my post entitled A Modest Proposal: VBScript.NET, aka "Helping Mort use .NET".

 

It’s been one hell of a week!

It has been one hell of a week, as the title suggests!  After getting back from VSLive I had over 300 emails in my inbox after the spam filter did its job, most of which required much more than just a read and delete. Hopefully this weekend I’ll catch up enough to start posting again. I’ve got a ton of things on my mind.

Oh, I met my blogging friend Eric Lippert at VSLive, as he mentioned on his blog. Nice to put a face with a blog.

Optimal Dressing, or A challenge to Eric Lippert

Ah the memories

Eric Lippert’s blog post about partial order sorting was, as always, interesting and well written. However, this specific post brought back memories of a project I did for Gateway Foods of Lacrosse Wisconsin back in the early 90’s. (I googled them and they don’t appear to no longer exist.)

Though my project was not as fun or easy to understand as Eric’s example, Gateway wanted to perform the "hot and new" business analysis called Activity-Based Costing, or ABC. Instead of just looking at which food items were most profitable, they looked at which customers were most profitable.

Aced-out by the SQL dude

They hired one of the big eight firms (remember when they were called "the big eight?) Booze-Allen-Hamilton to do the analysis and then went looking for a programmer. I heard about the project from my contact "George" for whom I had delivered a training course on Clipper earlier. I got excited, but really had no chance at it because the IT department didn’t believe in PCs, so they hired a SQL programmer to implement.

A few months went by and George called to tell me the project was going no where. A few more later George cakked to tell me the project was in real trouble. A few month more still, and I got a call from George asking if I could I start on the project that week in hopes of salvaging it (of course they had already spent most of the budget on the SQL guy, but what can you do?)

Be careful what you wish for

Later that week I arrived and met with the people from Booze, and they proceeded to hand me a stack of paper 10 inches thick that contained the allocation formulas! (I kid you not.) My job was to write a Clipper program to apply those formulas to a HUGE 300Mb database (remember, this was the early 90s and 300Mb was huge back then.) The allocation formulas were for things like "Allocate the cost of fuel to deliver to each customer based on the miles required to travel to customer. Sum the cost of fuel and multiply by the miles traveled for each customer divided by total miles traveled to all customers." And so on. Around 1800+ formulas in all.

For the next three days, my eyes glazed over as they tried to understand the formulas. I starting thinking "What the hell have I got myself into? There is no way I will ever be able to understand all of these formulas well enough to encode them into a program. No wonder the SQL guy failed. I am screwed too!" But finally, it started to dawn on me; "Don’t learn the formulas, learn the patterns." I’ve always been extremely good at pattern recognition so why didn’t I see it immediately? (which contrasts nicely with all the other things at which I’m extremely bad.)

Patterns; what a concept!

The initial database had number tables like fooditems, customers, and order. It turned out there were only three patterns:

  1. Scan a table and perform a calculation like sum or average and write the output to a one-record table. Examples included calculating total sales, average order size, etc.
     
  2. Scan a table and perform a calculation like sum or average for each fooditem, or customer, or similar, and write the output to a table that has one record for every fooditem, customer, or similar.
     
  3. Scan a table and perform a calculation like every record and write the output to a table with the same number of records as the scanned table.

That’s it. Three patterns. All 1800+ formulas boiled down to three patterns. SQL gurus know the following are examples of the above three patterns:

  1.    SELECT
            SUM(SubTotal) AS TotalPrice
       INTO
            OneRec
       FROM
            Orders
  2.    SELECT
            CustomerID,
            SUM(SubTotal) AS TotalOrdered
       INTO
            OnePerCustomer
       FROM
            Orders
       GROUP BY
            CustomerID
  3.    SELECT
            opc.CustomerID,
            opc.TotalOrdered/c.TotalDeliveryMiles AS SalesPerMile
       INTO
            CustomerFactors
       FROM
            Customer c INNER JOIN
            OnePerCustomer opc ON opc.CustomerID=c.ID

Technique triumphs over toolset

Ultimately I succeeded using Clipper where the SQL programmer had failed. He didn’t fail because he used SQL and my success was not because I used Clipper. He failed because he tried to hand-code all 1800+ formulas and I succeeded because I wrote a Clipper program to handle the three patterns, not the 1800+ formulas. In hindsight given the tools available at the time, the best solution would probably have used Turbo Pascal and SQL, but such is life.

The Gateway people entered the allocation formulas from Booze into a grid in my program, my program then determined which formulas to calculate and in which order, and then program executed those formulas. I used Clipper’s "macro" capability to allow me to execute the text formulas much like how Execute() can be used in VBScript. However, VB.NET programmer could do the same thing today by writing a program that generates SQL and then passing that SQL to ADO.NET to execute.

Back to Eric’s Post

The "which formulas in which order" was the part I was reminded of by Eric’s post. It actually took me three weeks to write that first program. 12 hours a day. 7 days a week. In the dead of winter. In Lacrosse Wisconsin. Brrrrr. (I’m from the south in Atlanta and I am a wuss when it comes to frigid weather!) Today I could probably write that same program in VB.NET and SQL in an afternoon. But then hardware is a lot faster now.

Intermediate Representations

Why three weeks? My program took almost two days to apply the 1800+ formulas to the 300Mb database! Ever try to debug a program where you have to step through code for 24 hours before you get to see the next bug? Not very productive. It was here I actually learned the tremendous value of creating an "intermediate representation." HTML is a perfect example: one program can generate HTML and another can display it. SQL, XML, and MSIL are also immediate representations. One of the benefits of an intermediate representation is you can validate output without having to observe behavior. In essense intermediate representations provide much the same benefits as state machines.

I split my program into two modules; one that compiled the formulas, and a second that executed the compiler’s output. My "compiler" took only about 2 hours to run which allowed me to debug my program in this lifetime. It still took two days to execute the formulas, but at least those formulas executed correctly!

Just when I thought I was out…

I went home victorious. Or so I thought. A few weeks later I get a call. (They tracked me down while I was on vacation, no less!) The Gateway people decided they wanted to modify the formulas and run "what-ifs." After each "what-if" it took them two days to get an answer. "Couldn’t I make this thing any faster?" (Imagine if the SQL guy had succeeded in encoding all those formulas into a SQL script? He would have had a job for life! A boring-as-hell job, but job nonetheless. Until Gateway realized all the money they were wasting and then threw him out on his ear.)

My original program, except for the compiler part, was almost exactly what Eric Lippert blogged about a few days ago. I was damn proud I wrote it. However, optimizing it was much, much harder. Turns out what was happening was each pass on each of the tables took a really, really long time (we knew that), but also there were formulas that could be performed on the same pass, but I doing a complete pass for each formula.

Well, to finally cut a long story short, another three weeks and I was able to implement an optimizer which batched together formulas that could run at the same. That version took only about 6 hours to run. Good enough, and that was the last of Gateway foods for me (I never was very good at milking clients; guess that’s why never made much money while in consulting.)

Optimal Dressing

So, back to Eric’s example. Let’s assume it took one minute for each seperate Eric put on, but if he was able to put on multiple items "at the same time", then each batch of items would only takes him 1.5 minutes to put on. His original program’s output, which dressed him in this manner:

      tophat      : [ ]
      shirt       : [ ]
      bowtie      : ["ashirt"a]
      socks       : [ ]
      vest        : ["ashirt"a]
      pocketwatch : ["avest"a]
      underpants  : [ ]
      trousers    : ["aunderpants"a]
      shoes       : ["atrousers"a, "asocks"a]
      cufflinks   : ["ashirt"a]
      gloves      : [ ]
      tailcoat    : ["avest"a]

This would take Eric 12 minutes to dress. However, if he batched items that could be done at the same time (i.e. items that had no prior dependencies), Eric could dress in 4.5 minutes:

      tophat      : [ ]
      socks       : [ ]
      shirt       : [ ]
      underpants  : [ ]
      gloves      : [ ]

  then

      trousers    : ["aunderpants"a]
      bowtie      : ["ashirt"a]
      vest        : ["ashirt"a]
      cufflinks   : ["ashirt"a]

  then

      pocketwatch : ["avest"a]
      shoes       : ["atrousers"a, "asocks"a]
      tailcoat    : ["avest"a]

The Challenge:

So here’s my challenge Eric (and I hope you don’t hate me for it): Implement an optimizer that will generate batched output where items in each batch have no interrelated dependencies, as shown in the last example above. It was a bitch to write in Clipper; how about in JScript?