Friday, April 23, 2010

Agile Estimation and Planning: Demystifying the Black Art

This article is not about how to do estimation and planning in detail for Scrum teams, no magic recipes or silver bullets, it´s not also about numerical models or techniques one hundred percent bullet proof. This is more about what I saw that it works in practice and how this correlates with Agile and Scrum values.

That been said and if still have your attention, let me start saying that I´ve borrowed the phrase "Demystifying the Black Art" from a Steve McConnell´s book. I did this for a very good reason, I´ve read that book many times and my only conclusion is that is impossible to provide accurate estimations for the simple reason that no one can predict the future. Understanding this simple fact prepare us for start doing Scrum estimations. Part of the preparation is, if you´re a manager, that you´ll need to resign to have everything accurately estimated before work even started.

But then what? Are Scrum projects totally chaotic? In a way, yes they are, the great difference though is that Scrum publicly recognizes that chaos is there, embraced it and prepares your mind set to cope with continuous change and adaptation.

Reestimate, Reestimate, ….

So, the new game becomes now not how good am I estimating, but rather how often I reestimate. Estimations in Scrum are not cemented deals, they are negotiable as work progresses through sprints, but then, how I can keep things under control without exhausting my team by adding more and more work? Answer is simple; if you add work then you move less priority work to the backlog and reestimate accordingly.

Again, is not a matter of having precise estimations, is a matter of being prepared to reestimate whenever is necessary. Further, there´s a learning curve that Scrum take full advantage of it. For instance, if you estimate a task and then keep your estimations even if you know they are not longer valid, then there´s no chance to refine them with new information and experience. Scrum on the contrary favors that you look and your estimations and make them up as many times as you feel so.

Last part sounds like nightmare for many managers, but come on, reestimating not only means that the number of hours will increase; frequently reestimations take things in the other direction. After all, everybody knows that people try to be over-conservative when providing initial estimations. So if you reestimate, there´s a fair good chance that you reduce the initial slack that you put just in case things get ugly.

No Black Art Anymore

But wait, estimation is still black art, something is missing and that is how to actually estimate. You can read many good books on the subject written by Mike Cohn, even attend to his seminars if you can afford it, but here´s is my simple and free advice: use group estimation. Estimations that are provided not only by one individual but by a group have more value and more chances to be not so much off the mark, why? Only because group estimations force the group to think it through before putting estimates.

Another piece of free advice, actually this comes from McConnell, estimate using ranges like thinking first in worst case estimates, then in best case estimates and put your more realistic estimate somewhere inside that range. Again, if you think in this as a group exercise you´d have more chances to make it successful, at least it won´t be boring :)

But what if the group can´t come to an agreement? Well you can buy the Cohn´s planning poker decks; note that you´ll need one deck per each individual in your team. I offer you a cheaper and in some cultures more fun solution: just vote using your fingers to indicate numbers, like one finger for one, two fingers for two and so on.

Team Alignment is Key

If we reach this far you’re almost ready to discover the secret behind planning: the truth is that Scrum planning is not more than a short meeting (by short I mean less than four hours) divided in two halves whereas the first half is for understanding what user stories are and the second for providing estimations. Again nothing new here, Scrum has been talking about planning meetings for quite a while now. Understanding what to do seems to be crucial, but wait we´re doing Scrum, aren’t we? Don´t fall in the trap of trying to understand what needs to be done to the last detail even before coding a single line, the only way to truly understand something is when you actually do the work.

What´s the value of having planning meetings? The answer is quiet simple: you get all your team in room and make them talk about their understanding of what needs to be done. That simple exercise will do the magic of aligning everybody´s energy and attention towards a common goal (a.k.a. Sprint goal). Like Jeff Sutherland said "Scrum is all about aligning your team in the same direction" and hence is the true value of planning meetings.


  1. Hi Juan, I like your commentary on re-estimating. I find that my product owners (I am a UPO with 3 POs) are sometimes surprised when they take a set of stories that were roughly sized a few months ago and try to add them to a sprint. The team wants to split and re-size which often results in a larger point total. I believe this is normal but it is difficult to convince one very point-oriented PO of this. Have you seen this on your team(s)?

  2. Hi there, awesome site. I thought the topics you posted on were very interesting. I tried to add your RSS to my feed reader and it a few. take a look at it, hopefully I can add you and follow.

  3. "The team wants to split and re-size which often results in a larger point total. I believe this is normal but it is difficult to convince one very point-oriented PO of this. Have you seen this on your team(s)? "

    Well this is a common situation in Agile projects, there´re several dynamics occurring that triggers the need for re estimating. More often than not, re estimation implies adding points to the original estimates which is normal considering the degree of expertise that the team is acquiring as it progresses.

    What it worked fine for me was presenting these facts to POs like a more refined version of reality, without making-up things or trying to hold on unrealistic estimations. Usually POs have shock reactions but this is mitigated because the earlier that they know the better they can accommodate.

    Again, the principle is very simple: provide as much transparency as you can, even if you have to say that you´re estimations were completely offset. This transparency is what will allow the team and the PO to react in time.

  4. Regarding the value of the planning, not just the extent, this seems to be in question in many environments. I've tried to summarize some of the reasons to perform task-level estimation, along with some techniques: