Predictable failure demand

This blog I want to discuss how we handle defects coming from the system testing department (where the complete system - hardware and software - is integrated). 

How many defects can we expect during development? And more important, how many defects will be submitted after we have implemented all Stories? After all we can estimate the work we know that is coming (Stories) but how to anticipate on future defects?

Since it was not clear how much time we should reserve for defect solving, and hence how much time would be left for implementing stories we struggled how to fit these defects into our sprint development activities. Because of the nature of defects some teams proposed to go to Kanban (at least with a part of the team). Personally I would like to solve the issues you find instead of a finding a way to live with them.

Defect Prediction

Some years ago I had a discussion with people from the company QSM which claimed that you can predict the amount of defects in a software system quite easily. Just summarize the amount of defects submitted for a certain product for a chosen time period, plot them in time and match them with a Gaussian curve. I decided to test this years ago with a running project. At that time nobody was interested but years later I remembered the exercise and thought it would be interesting to complete the experiment and see if the prediction was any good. I was completely astonished, apparently I was able to predict the amount of defects for 14 months ahead within 1.5% accuracy. It was a large project and it involved thousands of defects. I simply could not believe it. So I decided to check another project. Exactly the same result.

Look at the diagram below. Believe me, no tricks are performed on this data. The solid line are the amount of defects submitted for a specific product each month (from cradle to final delivery). The dotted line is the parabolic trendline. 

  

Some notes on this:

• As expected the parabolic curve reaches its top after all functionality is implemented. There are no more Stories left to implement and we are in 'maintenance' mode. After this moment the curve becomes more reliable (the landing point is rather stable). 
• You notice that the amount of defects that can be expected during the development time of a project is not random. In fact it follows a line and is quite predictable. 
• The trendline is not followed fluently. You'll see a lot of hick-ups. I think it is due to the fact that we the system testing is performed on regular releases of the software. This causes a peak of defect submission just after the release (and dips when the release is maturing). You'll get a peak of defects after each releases of which the testers expect fast repair times. This implies a lot of 'disturbance' in your sprint. It would be better to decrease the time between the releases. Or the release (and test) continuously ;-) 

Follow the line

In order to follow the trendline we introduced a simple rule "the minimal number of defects that is solved in sprint <n> is the amount of defects submitted in sprint <n-1>”. Note that not all defects found in the previous sprint need to be solved in the next sprint - only the amount. This approach has the advantage that we solve defects as fast as possible which is good from the 'eliminating waste' point of view but also for the software engineers (whack the mole pattern). They get feedback as fast as possible and can still remember the code. 

We do not estimate defects (no story points), we just follow the line. There is no reason to estimate defects since - we decided - they do not contribute to the velocity. Our velocity indicates how much work we can develop correctly each sprint. Stories are estimated, and this estimation covers a correct implementation of the work including possible defect-solving. 

One word on quality. Focus for all Agile teams is quality. I see this fact underestimated time and again. Loosing quality implies a lot of disturbance and your velocity will drop instantly (besides other negative effects like unhappy customers, cost increase, unpredictability, etc.).

Predictable failure demand

Since we do not include defects in our velocity, the sprint capacity will drop when working on defects. Because we have the prediction line we can actually predict what the velocity will be for future sprints (velocity - number of expected defects / 6). This is what we call predictable failure demand. With this information we can more accurately estimate the impact on future release.

Embed defects in the sprint

It is easy to say to follow the line but actually realizing this can be quite hard. How to incorporate the defects in the sprint? For this we’ll assume a constant average repair time for each defect and that 6 defects represent 1 Story Point (actually we based this assumption on estimations based on hours). Now we can exchange Stories with defects. 

To prevent large sprint backlogs we bundle the defects in blocks. Each containing multiple defects. The defect blocks can be estimated and individually prioritized in the sprint backlog. Most of the time we have 2 defect blocks. Defect Block 1 has the highest priority and is on top of the backlog (so first solve problems before making new ones). Defect Block 2 never has the lowest priority to prevent that defect solving does not make it until the end of the sprint. 

As a metric we keep track of the ratio between solving defects and realizing Stories. Target is that we should be working on Stories for more than 90% of the time. If the amount of defects is too large (the trendline becomes steeper) we take action immediately.

That is basically how we integrate working with defects into our sprint. By doing this we are capable of performing a good and reliable sprint planning and to minimize the ‘waste’ to keep our customers happy. And, also important, take immediate action if things are not going in the direction as expected.

I would like to hear you responses on this topic.