Tom's Ten Data Tips - November 2011

Agile planning

Planning is a very important, and often undervalued part of Agile software development methodologies. It is eminently lightweight, and based firmly on empirical data as well as scientifically validated estimation methods. And in the Agile spirit, investments (planning takes time) are postponed as long as possible in favor of delivering value early and often, and gathering more valid data to underpin delivery plans.

The way you manage a software project builds in large part on “conventional” project management. However, there are important differences. Why is a programming effort so hard to manage? Because coherent design is so important, software projects become exponentially more difficult as they grow in scope and complexity. By their nature, computers must be instructed perfectly or else they’ll fail. Finding the first bugs is easy, but the last ones become exceedingly hard to locate and fix. That’s why there’s a premium on writing quality code.

1. Planning Is An Integral Part Of Agile Practices

Often, Agile developers are “accused” of not doing any planning. But just because Agile practices advise against a comprehensive, detailed specification of every step along the entire project upfront, this doesn’t mean that no planning is involved. To the contrary. Agile planning is focused more on the planning (verb) than on the creation of a plan (noun).

Agile planning encourages change, which is accounted for in both the plans themselves as well as when they’re made. Planning is spread out more evenly throughout the project, and done in a “Just-in-Time” fashion.

2. Planning Is A Quest For Value

In most organizations, planning is “simply” demanded because activities across business lines need to be coordinated. For the purpose of teamwork and delivering value as a concerted effort there is usually ample opportunity for agility and flexibility in plans.

The deeper Agile purpose of planning is quite different, though. The Agile, ongoing Just-in-Time and iterative approach to planning serves to constantly deliver the most possible value in the short-term. Overall this adds up to the most value over the lifetime of the project. If business partners are to be involved in this activity, they need to know how much it costs to build certain features, how long it will take, what the impact might be on the architecture, etc. And all these considerations need to be presented in a non-technical way so that all participants have an equal opportunity to contribute to discussions.

3. Agile Planning Happens Throughout The Project (Instead Of Upfront)

In a Waterfall project, plans are made at the outset. Functional needs are translated into technical detail specifications. A comprehensive inventory of all the tasks foreseen is then turned into technical specifications. This allows detailing a plan before any work has begun.

In Agile projects we don’t ‘believe’ in these plans for three reasons: the best architectures emerge as you are building and learn more about the final implementation. Secondly, the actual product never perfectly matches the upfront plans, and then a “game” of push and pull tends to begin that jeopardizes trust and cooperation between business owners and the development team. And thirdly, the plan itself delivers no value to the business. It is hypothetical to begin with, so we’d rather start out building the most valuable features first and then build and deliver incrementally to produce value for our clients as early as we can. And that implies doing ongoing planning “Just-in-Time” as needed to support day-to-day decision-making (but no more).

4. Agile Plans Are Based On What We Know (And Not What We Hope…)

A central component to Agile planning is acknowledging and accepting uncertainty. In Agile methods we look this uncertainty straight in the eye, and make no attempt to deny it. No amount of upfront planning that is based on hypotheses (as opposed to empirically grounded observations) will defy inherent uncertainty. You can “make believe” it doesn’t exist in a plan, but the actual course of events will then become a roulette. The casino is a much nicer place to play gamble than a software shop.

5. Estimate Size, Derive Duration

The rate of progress on Agile projects is measured by its “velocity.” Velocity can be expressed in unitless story points or perfect engineering hours. Story points or perfect engineering hours are a measure of feature size and complexity. Cohn (2005) “A key tenet of agile estimating and planning is that we estimate size but derive duration.”

The team (it is a joint effort) estimates the number of story points (or perfect engineering hours) it assigns to a feature. Based on historic velocity, the time it will take to deliver this feature (or set of features) is then calculated (derived). It is simply analyzed on the basis of past and thus proven performance.

6. Change Management Shouldn’t Become Change Resistance

“Embracing change” is a core Agile principle. The agile manifesto reads: “Responding to change over following a plan.” We embrace change for the customer’s competitive advantage.

However, when you “scale out” Agile to larger teams and a corporate environment, invariably some “change management” process becomes ingrained. When anybody can instruct developers to change their activities on any given moment, this will harm productivity (in Scrum these changes are funneled through the Product Owner, of course). Unfortunately, formal change management processes quickly take on “change resistance” properties. This goes directly against the Agile grain. So beware that “change management” only serves to improve productivity, and be willing and free to challenge any formal process in Retrospectives if you doubt it serves the customers’ best interests.

7. Design Your Agile Plans With Adaptation In Mind

One of the prejudices against Agile is that Agile teams don’t plan nor commit to dates. This critique does injustice to methodic, scientifically underpinned planning practices that have come to the fore in the Agile community. Because Agile methods are meant to be, well, agile, …, there is an inherent bias towards making plans and estimates that optimize adaptability rather than putting your hopes up that the release will be built exactly as, and in the order foreseen.

By making iterations as independent as is reasonably possible, we provide the business with options to choose their preferred order, and we enable (and allow) them to change that preference at limited cost. Because if changes are prohibitively expensive (e.g.: a lot of work gets “wasted”) we’re not really offering a choice, nor are we empowering the business to leverage their freedom to make changes any time they see fit.

8. Accelerate Development Of Features That Generate Product Or Project Knowledge

An important part of Agile planning is the explicit acknowledgement of uncertainty not only of what you are going to build, but also how. Therefore, beside the obvious need to release the most valuable features first, we place a premium on building some parts early that will help us manage risk and generate learning that is needed to further develop and refine the overall release plan. If a more accurate planning is of great importance, you can meet that need by scheduling some difficult or particularly risky parts first to reduce some of the uncertainty in the planning.

9. Plan Based On Features, Not Tasks

A fundamental difference between Agile planning and Waterfall plans is the level of granularity at which plans are made and managed. In Waterfall you typically use Gantt or PERT charts to plan for work, and the work is planned based on individual tasks that have been identified as necessary to build a feature. In Agile however, the planning is done at the feature level and if this is broken down into tasks, these are ‘only’ used to monitor progress.

By planning at the feature level, you push down authority to organize and decompose the work to the person actually doing it. This gives them maximum leverage to think of alternative ways to get it done. They can also manage their own work independently as they see fit. Degrees of freedom tap into the creativity and entrepreneurial spirit of the individual, hence increasing productivity. By “only” planning at the feature rather than the task level, you take another step in “maximizing the work not done” – another productivity booster.

10. Good Enough Planning Is (Usually) Good Enough

Planning is not an exact science, and unlikely to become one anytime soon. This holds for Agile planning, and other software development methods alike. Agile planning is deliberately meant to stay lightweight because we always try to minimize upfront investments in favor of producing valuable products early and often. There are two important reasons to settle for a “good enough” planning. Firstly, the difference in accuracy between a quickly conceived planning (which experienced teams can do in a matter of hours), and a more elaborate and detailed one is rarely worth the extra time. Some say it can actually become worse with more upfront effort (e.g.: Cohn, 2005).

A more detailed plan simply doesn’t become much better because of big uncertainties embedded in it. And it would (or might) suggest a false level of precision. But even if the effort does lead to a better plan, the difference between a good enough plan and a refined one probably isn’t worth the additional effort. The plan only needs to be good enough so that it can be used for making sensible choices to maximize the progress of the project. In Agile we promote transparency, so setting improper expectations is something we should avoid. Secondly, in Agile we embrace (and expect) change. Therefore, a more elaborate planning isn’t justified because we don’t (really) expect the plan to materialize as conceived. It would be a shame to invest heavily in something that immediately looses its value when business priorities shift. And they do often shift…

Further reading

Some excellent books on Agile planning:

Agile Planning and Estimation.
Mike Cohn (2005)
ISBN# 0131479415

Planning Extreme Programming.
Kent Beck & Martin Fowler (2000)
ISBN# 0201710919