Multi-Stage Continuous Integration Part I

I’m a big fan of continuous integration. I’ve used it and recommended it for many years. But it has a dark side as well. When it is combined with the practice of mainline development, especially for a large development effort, it can turn into “Continuous Noise.” In this case, you get notified every 10 minutes or so (depending on how much building and testing is going on) that the build and/or test suite is still failing.


In the diagram, the stability of the mainline is graphed over time. In this example, the graph starts when the product is released and the mainline is 100% stable. It builds without errors and all tests pass. As new development starts, developers make changes that break the build or cause tests to fail. Because they are all working against the mainline, it is likely that just as one developer fixes the problems they caused, another developer is checking in more changes that will create one or more new problems.

As the stability of mainline goes up and down, everybody that is working against mainline is affected. If the mainline doesn't build or there are tests that don't pass, then everybody has to wait until the problem is fixed.

It could be argued that the problem in this case is that the developer that broke the build should have updated their workspace, done a full build, and then run the full test suite after that. Basically, a full integration build and test. In practice this doesn’t work very well.

For any meaningful project, a full integration build and test will take time. Even if you somehow get that time down to 10 minutes, a lot can happen in 10 minutes, such as other people checking in their changes and invalidating your results (whether you realize it or not). It is more likely that this full cycle will take on the order of an hour or more. While you can certainly do something else during that hour, it is inconvenient to wait and necessitates task switching which is a well-known productivity killer.

Even though developer integration build and test has its problems, many shops have implemented it. While this practice is not ideal, it is an idea that is headed in the right direction.

Next: Multi Stage CI Part II .

There is no Bug. It is not the Bug that Bends, it is Only Yourself

In the movie “The Matrix,” when Neo goes to visit the Oracle, he has a conversation with one of the potentials who is bending a spoon with his mind. The boy explains to Neo that “there is no spoon” and that “It is not the spoon that bends, it is only yourself.” While this is rather mystical, it occurred to me that this is a good way to think about bugs!

Often, getting a bug to bend to your will (go away), is about as easy as bending a spoon with your mind. For instance, consider the problem of coloring maps such that no two contiguous regions are the same color. Let’s say that in your experience, you’ve only ever needed 3 colors. You need to color maps using software, so you write it with the assumption that you only ever need 3 colors. You may encounter some bugs which end up putting the same color on two contiguous regions in a map that can be colored with just 3 colors. Eventually you will find and fix those bugs. But if you run your program on a map that requires 4 colors, no amount of bug fixing your program that uses 3 colors will ever work. You must realize that your assumption that 3 colors will always work is wrong.

Another problem with bugs is that sometimes fixing the bug doesn’t actually fix the root problem. You think you know what the bug is, so you make a change so that the software now behaves as desired under the given circumstances. However, there are still 5 other ways that the problem can manifest itself and you didn’t fix any of those. Plus, you’ve introduced a new problem. You don’t know about these 6 problems yet, but they are there. You’ve only fixed the symptom, not the cause. You have not bent the spoon.

Even when you find and fix the root cause of that bug, you will write more software using your existing process and it will have new bugs to replace the old bugs. The progress was an illusion. The spoon is still unbent. If anything, it is even more unyielding than before.

It is tempting to think of a bug as something that creeps into the software after it is written, something that is separate that can be isolated and removed. But really, bugs are introduced during the process of developing software. They are not separate and isolated, they are actually flaws in the process itself. The problems lie within. Only when we acknowledge this ("there is no bug") and accept it can we hope to affect real change. Agile development is an excellent platform for seamlessly incorporating the required introspection and continual improvement.

I’m not saying that real software with zero bugs is an attainable goal, only that you consider a change of perspective in order to bend the spoon a bit. The ultimate goal is to make behavioral and process changes such that no bug is found in the software that you deliver.

Next: Apply Elegant Architecture to Your Dev Team

What is Your Impression of Agile?

I'm interested in hearing people's first and/or current impressions of Agile development. Here's mine.

My first real encounter with Agile development was at the Software Development Best Practices conference in September of 2005. Up until then I had managed to be all but ignorant of it. I was at the speaker’s reception, and unbeknownst to me I had been dropped into a den of Agilistas. They all looked like perfectly normal people to me. Conversely, they all seemed to assume that I was “one of them.” But then they started talking to me about 3x5 cards, pair programming, collocation and the like.

I wondered why these crazy people thought it would be a good idea to send poorly tested software to customers on a weekly basis. And what were they thinking doing their planning by shuffling around 3x5 cards with notes scratched on them? These folks were talking about writing software while sweating hip to hip with a fellow programmer all day as part of a two-person programming team in a small cramped space in close proximity with lots of other sweaty two person teams. And when I objected I was treated as a heretic that “just doesn’t get it.” To top it all off, this stuff is all written up in a Manifesto!!

I wanted to run as far as I could as fast as I could. In talking to other people about their first contact with Agile, I have found that this initial negative reaction is quite common. My reaction was so strong I decided to put together a whitepaper pointing out all of the problems with Agile and introducing an alternative to Agile which could be loosely described as “process and tools done right.” I felt that Agile, and especially Extreme Programming, was an overreaction to unwieldy project management, process, and tools. The more research I did for the paper, the more I found to dislike.

Along the way, I finally realized why Agile is a good thing. But, it took me close to six months of active research motivated by my dislike of Agile to see the light.

There really is something significantly new, different, and worthwhile lurking in there and it doesn't require small teams, pair programming, working in an open area, 3x5 cards, or fanaticism. This blog is an attempt to examine the benefits and inner workings of Agile from a technical perspective. That is, what exactly are the benefits, how exactly are they achieved, and why is it different than traditional development.

Describing Agile is a bit difficult, let's take a look at it at a high level to start with.

Next: Agile in a Nutshell

Agile Development is People Oriented

[This post is a remix of an earlier post and also a repost from my WordPress blog]

There’s a lot of literature available on the usability of both software and everyday objects. But have you ever considered the usability of the process of software development itself? Do you feel like you know exactly what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next?

In my experience, traditional software development involving long iterations suffers from poor usability and hinders the ability of otherwise intelligent people to get their work done. On the other hand, there is something special and people-oriented about Agile: short iterations leverage human nature and thus allow us to fall into a more natural rhythm in which we make fewer mistakes and work more productively.

While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, people are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Consider the following statements. I’ll bet that you’ll agree with most of them.

• When working on software for personal use, people make changes and then use the new version right away.

• One of the things that developers enjoy about software development is the fact that they can make a change and see the result right away.

• When working on a hotfix, people are more likely to cross all of the “t”s and dot all of the “i”s and take extra care to make sure that everything is done just right.

• It isn’t easy to create something that the customer thinks is exactly right the first time.

• People prefer to delay making important decisions as long as possible.

• Towards the end of a release when a new bug is found or a great idea for an enhancement comes in, it is tempting to just put it into the current release rather than the next release because the next release won’t happen for quite a while.

Each of these is a part of human nature. All of the positive points are reinforced with short iterations, and all of the negative ones are reduced with short iterations. Conversely, the exact opposite is true of long iterations. Long iterations go against the grain of human nature and thus create usability problems.

Next: The Agile Waterfall

Desiging Software is the same as Predicting the Future

Software Design Basics
Designing software requires a myriad of skills and knowledge to do well including: domain knowledge, market knowledge, technical aptitude, up-to-date knowledge of the technology relevant to the domain, and the ability to spot patterns and trends. If you have all of these, you have the basics required for the knowledge-based decisions that you must make as you design your software. But designing software also involves dealing with the unknown.

Predicting the Future
It is impossible to know exactly how many customers or users you will have, exactly what features will bring in the most revenue, which implementation technologies will work the best for you, the effective useful life of various implementation technologies, or which of many possible implementations will best satisfy your users. For these factors, you must make guesses. In other words, you must predict the future!

Predicting the future is hard. The more information you gather and the more variables you consider, the better your prediction will be, but the more time your prediction will take. Also, the more possibilities you take into account when you design your software, the bigger and more complex your design will become and the longer it will take you to produce the design. The bigger and more complex your design becomes, the more likely it is that you will have difficult design problems to solve. Solving difficult design problems can take a long time. Once you've finished this big and complex design, it will take a proportionately long time to implement that design.

When you put all of this together it is obvious that the more accurately you try to predict the future, the more time and effort it will take. The question is, how do you decide when your design is good enough? If you take it too far, you'll either never finish your design, or you'll never ship your software. But if you don't spend enough time on design, you may end up with unhappy users or software that needs to be rewritten.

Consider the problem of predicting the weather. Modern forecasting software is pretty good, but the further out you look, the longer it takes for the forecast to run and the more likely it is for the forecast to finish after the day you are forecasting has already come and gone. The other factor to consider when designing software is that technologies and user needs change almost as frequently as the weather.

Iterative Design and Iterative Implementation
What can we do to solve this dilemma? There are two choices: spend lots and lots of time on up-front design and the software that results from that large design or do iterative design on a smaller set of features. When you do big up-front design for a large set of features, it is inevitable that you will have to rewrite large parts of your software in the future to take into account changes in technology or user needs. Of course, you don't have to accommodate change, but then it is also less likely that you will maintain or increase your revenue stream. Since it is inevitable that you will have to make changes to keep your software relevant, why not just accept this fact and start building the skills and implementing the practices to get good at iterative design and iterative implementation? The alternative is that when you are forced to change your software, it wasn't written with change in mind and your team isn't used to making changes on a regular basis.

It isn't that hard to make this decision. After all, everybody knows that predicting the future is hard. What's more important, getting good at predicting the future, or getting good at adapting to change?

Next: The Simplest Thing That Could Possibly Work