Tom's Ten Data Tips - April 2011

Surrounding requirements

Requirements form the beginning of any (software) project: someone has a request that needs clarifying in order to get the project of to a good start. But producing valuable BI solutions from vague requirements, and often using unstable technology is very hard work!

“Gathering requirements” is often used for this phase, but we don’t particularly like that connotation: as if there are grapes hanging from the vines, ready to be picked. Our practice has been quite different. If users know and can formulate their requirements at all, they are vague and mostly pre-conscious. Rarely ripe and ready to be harvested…

1. Surrounding Requirements Begins With Understanding Key Business Drivers

An enterprise data warehouse needs to support key information needs throughout the corporation. And for departmental projects just the same holds for this business unit. To prioritize, and sort the chaff from the wheat, it is crucial that information analysts be thoroughly familiar with corporate strategy. “Data warehouse designers must understand the key factors driving the business to effectively determine business requirements and translate them into design considerations.” (Kimball et al, 2008).

Unfortunately, there is no ‘obvious’ place to look for these “key business factors.” If a comprehensive and explicit corporate strategy is in place, it should provide this input (just don’t count on it). You can never accept the corporate portal with PR blurb at face value. More likely senior managers’ personal targets provide a clue. But keep looking and challenge “conventional wisdom.”

2. Agile Methods Need Requirements Analysis, Too

As the industry has gone through phases from ‘genuine’ waterfall via iterative methods to Agile approaches, the requirements process itself has become less and less defined. Sometimes there appears to be a (false!) belief that Agile development methods can move ahead while the requirements are sorted out later.

3. Requirements Are Never Complete, So Count On Ongoing Analysis

Regardless of what your functional or technical documentation looks like, you need to support ongoing research into needs of (end) users, because requirements are never complete. Often users only begin to figure out what they really want, after they see the first (working) prototype. Sometimes programmers who’ve been handed specifications are not expected to elaborate on requirements. Jeffries et al (2001): “The most effective way to understand a requirement is to discuss it.” A phrase like “just build it like this” smothers your developers’ initiative and creativity. It also displays either disdain for their contribution to the requirements process, or a total lack of experience.

Besides ‘feeding’ ongoing requirements research, you also need to employ overlapping development phases. Even Winston Royce (“Managing the Development of Large Software Systems”, 1970) himself (the author of the “classic” paper on Waterfall methods) was also quite aware of this need, and included it in his original paper. BI projects do not proceed in sequence like Waterfalls always flowing down. Force fitting such an approach looks a bit like denying gravity. It still exists.

4. Prototypes “Work”

In many cases, customers only really start to understand what they want, after seeing and interacting with a prototype. When you build a prototype, you provide a rendition of someone else’s impression of their needs. The opportunity to actually see it and interact with it, leads to a fuller expression of user requirements, and provides the necessary input to build a more valuable solution. Money and time spent on the prototype are never wasted; in fact they are an economic jumping board needed to do the best you can.

5. Requirements Are Rarely Omitted, But Often Unknown

When requirements appear to be changing (too) late in the development process, this causes rework and potentially redesign. It poses a huge problem in plan driven methods (Waterfall), but make no mistake: this harms efficiency in Agile or Lean methods, too. Of course Agile methods embrace change, even late in the development process.

Requirements documentation isn’t necessarily “wrong” per se, often there simply remain ‘some’ requirements out there waiting to be discovered. “Unknown” requirements here, stands for not knowable. At least not until fairly late in the design and architectural process, and potentially not knowable until end users begin to interact with (prototypes of) the system.

6. Usage Scenario’s Are (Usually) Easier To Surface Than System Goals

A tricky part of requirements engineering is making tacit knowledge explicit. Sometimes peoples’ behavior embodies such knowledge, but to transfer this to (e.g.) developers and architects, it needs to be made explicit. Requirements engineering has evolved over the years. In the 70’s it was largely about finding ‘the right’ way to model a system. Structured analysis and entity relationship diagrams stem from these efforts. Besides the “what” and “how” questions, later “why” questions came to the fore, which resulted in goal-based requirements gathering.

Unfortunately, it turns out that formulating goals “in abstracto” is exceedingly hard to do. Sometimes (often), describing the scenario’s (or use cases) which define “success” is considerably easier. By vicarious observation of such patterns, requirements can emerge. This is often referred to as scenario-based requirements elicitation (e.g.: van Lamsweerde, 2000). By running through hypothetical scenarios with envisioned systems, engineers get ample opportunities to probe for needs, and test their assumptions. Which of course helps to close the gap between ‘real’ and ‘perceived’ needs of end-users and stakeholders.

7. Overt Resistance Is Easier To Manage Than False Cooperation

Identifying and consulting with all relevant stakeholders is a thorny aspect of requirements analysis. Requirements work is challenging, and getting sufficient time (or other resources) from important stakeholders can be difficult (sometimes very difficult). Maybe the last project was a complete failure. This was none of your doing, of course ☺, but everybody felt their time was wasted. Sometimes people have “personal agenda’s.” And sometimes people simply resist change, or can even be hostile because they fear a new system will obviate or eliminate their job(s).

Resistance is usually born out of a lack of trust between business analyst and stakeholder. By discussing your own feelings, acknowledging your business partner’s feelings, and including the context of your work (why is this system built, anyway?), you can create a level playing field. That doesn’t guarantee cooperation or trust, it just gives everybody involved the best chance of spending their time wisely. When people pretend to cooperate, but really doubt your intentions or sincerity, you have a bigger and more difficult problem to manage. A lack of trust can haunt you by making project sponsors unhappy, by added scope (you ‘missed’ requirements that were never expressed, but still need to be included…), schedule delays because of “meeting conflicts”, etc. Operating in such a ‘vacuum’ is close to impossible.

8. Requirements And Design Are Interleaved

In theory, there’s a requirements stage that may be revisited (possibly repeatedly), and a separate design stage. What we see in practice is that these two aren’t nearly as separate as you’d think. This has everything to do with the process of reaching consensus on requirements (see also tip# 9), and the way that impacts your design considerations.

Also, in Agile methods we expect design to emerge, so to speak, during development. It simply isn’t possible to make too many design decisions upfront, without committing to poor choices. Sometimes optimal design become eminently clear once you are programming, and then you may need to revisit your initial architectural choices.

9. Surrounding Requirements Is Largely About Negotiating

One of the aspects that makes requirements engineering so challenging is that diverse stakeholders, each with their own backgrounds, concerns, perceptions, skill levels, and potentially conflicting viewpoints need to agree to a joint set of specifications that will guide development. And of course it is impossible to perfectly meet everyone’s needs. Most Agile methods deal with this challenge by assuming either a single Product Owner (as in Scrum), or in extreme Programming we say the customer always speaks with one voice.

If you’re directly working with multiple stakeholders you have a challenge: how do you facilitate a discussion among people who do not have equal information, operate from a (widely) different knowledge base, and have conflicting needs? And facilitate in such a way that they all feel the outcome is an equitable compromise for every participant? This requires the fine art of participatory decision making (Kaner, 2007).

10. Leverage Your Bandwidth And Fidelity

In order to facilitate ongoing high-quality communication between project sponsors, developers, architects, and end users, it is important to watch for two crucial ingredients that enable surrounding requirements: high communication bandwidth and fidelity.

You increase bandwidth, for instance, by using open office spaces, and ensuring that all resources needed are sufficiently available. This is why we need the product owner to be a full-time member of an Agile team. Fidelity (high signal to noise ratios) can be improved by helping people focus on the job at hand. People are terrible multi taskers, no matter how hard we try. So avoid distractions, multiple assignments, and task overloading. You also increase fidelity by insisting that Product Owners who join the team must have the authority to make priority decisions independently. Otherwise they cannot be part of the Agile team.

Further reading

Some excellent books on Requirements:

Data Warehouse Lifecycle Toolkit, 2nd Ed.
Ralph Kimball, Margy Ross, Warren Thornthwaite, Joy Mundy & Bob Becker (2008)
ISBN# 0470149779

Exploring Requirements – Quality Before Design.
Gerald Weinberg & Don Gauss (1989)
ISBN# 0932633730

Extreme Programming Installed.
Ron Jeffries, Ann Anderson & Chet Hendrickson (2001)
ISBN# 0201708426

"Structured Analysis for Requirements Definition"
D.T. Ross and K.E. Schoman
IEEE Transactions on Software Engineering, Vol. 3, No. 1, 1977, 6-15.

The Facilitator’s Guide to Participatory Decision-Making.
Sam Kaner (2007)
ISBN# 0787982660