The Software Development Tractor Pull

Why was Microsoft's Vista release so late? In a word (or three) "Overwhelming Software Complexity." As Ray Ozzie has said, "Complexity kills." He's right, but he doesn't go far enough. In my article "The Software Development Tractor Pull" I start the process of discussing this issue on our website and lay out some thoughts about how we begin to address the issue.

In my experience, managing complexity is a major issue for many software engineering groups.

An analogy that I like is that of a Tractor Pull, which is a kind of competition where the goal is to see whose tractor can pull the most weight the greatest distance.

In a tractor pull, the competitor's tractor is connected to a "sled." The sled has wheels in the rear, and a flat pan in the front. The sled also has a weight-box, that moves from the rear to the front of the sled as the sled is pulled by the tractor down the dirt track. When the tractor first starts pulling the sled, making progress is relatively easy. Of course, this is because the weight box is over the wheels, and the wheels roll relatively easily over the dirt.

However, for every inch of forward progress, the effort required to move the sled increases. After the sled has gone about three quarters of the distance, pulling it is very difficult, and the tractor may begin to slip its wheels.

By the end of the track, pulling the sled is almost impossible. At this point, the tractor is spinning its wheels at a high speed, the sled is rapidly slowing down, and the noise and fury and energy consumption of the tractor is through the roof!

Now, when the tractor is really pulling -- the throttle is wide open, smoke and flames are pouring from its exhaust stacks, and the engine is producing maximum power -- the wear and tear on the tractor is at its greatest.

This is why after relatively few runs most tractors have to have their engines and drive trains completely overhauled.

In fact, it's relatively common for the engine and/or the transmission to break apart during a particularly difficult pull. Then, the driver and his crew are left looking at a broken tractor and a pool of oil and busted parts on the ground.

Now, let me ask you. Do you see any parallels between a tractor pull and the software engineering work you are doing in your organization?

In my experience, software engineering in a start up, or on a truly new project, is much like the start of a tractor pull. The work is not complex and progress comes relatively quickly and easily.

However, as time goes by, the work becomes more and more difficult.

Often, but not always, this is due to an increase in the complexity of the work itself. With every bit of progress we make "down the track" the weight box moves forward a little more, and continuing to make progress is more difficult.

So, what's happening here?

What's happening, in my experience, is that the complexity of the software engineering work is increasing. As the complexity increases, more and more resources are required while projects take ever more time to complete.

"Windows is now so big and onerous because of the size of its code base, the size of its ecosystem and its insistence on compatibility with the legacy hardware and software, that it just slows everything down," observed David B. Yoffie, a professor at the Harvard Business School. "That's why a company like Apple has such an easier time of innovation."

Microsoft certainly understands the problem, the need to change and the potential long-term threat to its business from rivals like Apple, the free Linux operating system, and from companies like Google that distribute software as a service over the Internet.

In an internal memo last October, Ray Ozzie, chief technical officer, who joined Microsoft last year, wrote, "Complexity kills. It sucks the life out of developers, it makes products difficult to plan, build and test, it introduces security challenges and it causes end-user and administrator frustration."

Now, if the business strategy depends on having a software engineering function that is fast and therefore agile, and yet the software engineering group is slowing down while consuming more and more resources, is this a sustainable state of affairs?

I don't think that it is. I think it must lead to the situation where competitors, sooner or later, out-manuever us.

So what is the solution?

The direction of the solution is to become much more effective in our management of the complexity of our software engineering work.

Now, "We are much more effective in our managing the complexity of our work" is a perfectly reasonable objective, but how do we achieve it?

Sadly, in a short article such as this one, I can't give this topic the full attention it deserves. But what I can do is give you a high level outline of the process for achieving it.

First, we have to start by defining -- clearly and with lock-tight logic -- exactly what we mean by this objective. That is, we would build the logic that tells us -- and others -- exactly what conditions we must create such that we unavoidably achieve our high level objective.

The development of this logic cannot happen in a vacuum, however.

Look, we have to divorce ourselves of the notion that the software engineering group is somehow only connected to the greater organization through the "interface" pictured on the org chart. In reality, the software engineering group is intimately tied in with all of the other functions of the company, and so changes in the software engineering group do affect all other functions in the company. (See an upcoming article entitled "What Happens In Vegas DOES NOT Stay in Vegas" for more on this.)

And so, in the development of our logic, we must understand -- and be able to communicate to others -- that complexity in a software code base has a "dual nature" of sorts. That is, it's not entirely negative, and we have to clarify our understanding of this or we will run right into a huge (and legitimate) wall of resistance.

Said differently, we have to recognize when complexity in our software serves us, and when it does not.

As we develop our logic, we must also seek out and address a number of "negative branches." A negative branch is a problem that we predict will be created if we take a set of actions we are currently contemplating.

Fortunately, there is a well-defined process for recognizing and addressing negative branches. So, this need not block us, if we are willing to do the work.

Once we understand what is necessary -- what conditions we need to cause to exist -- we can shift our attention to the development of the implementation plan. And let's not fool ourselves here, there will be many obstacles to implementing the necessary changes.

Again, we have a great tool for planning the necessary changes. And we can overcome the obstacles -- no matter how great they may seem when we first discover them. But there are many other aspects that also need to be addressed in order to succeed with such a change.

Finally, when we have the reviewed implementation plan, we can begin the process of rolling it out. This will take some time, as we are talking about fundamental shifts in how people perceive and do their work.

In closing, let me just say this. I'm convinced that complexity in software development work is a major constraint for many software development groups. There are great ways to address this complexity and thereby to greatly improve flow through that organization.

However, implementing the changes necessary to improve flow on a lasting basis is not trivial. It takes time, effort and a genuine willingness to examine some deeply held beliefs.

As always, I would be grateful for any comments or feedback you wish to offer on these articles. Please contact me for a login on our site.

Sunday, January 14, 2007 in complexity | management  | Permalink |  Comments (0)