Iterative Development Resources

As a follow-up to my earlier entry on The Unchangeable Rules of Software Change, one of the things that particular team figured out it needed was to try and use iterative development as a means of managing change and being more responsive to customer feedback while still controlling scope.

Of course, saying "do iterative development" is one thing. Figuring out how to actually do it for a group in an organization that isn't accustomed to it is another thing entirely. So here is a list of resources on the subject of adopting, planning/managing, and doing iterative software development -- particularly for those coming from a background of phased-sequential (waterfall,V) model of planning.

Iterative Development Resources:

• Wikipedia page on Iterative Development


• Iterative vs. waterfall software development: why don't companies get it? , Bill Walton, Computer World, Feb 2004


• Craig Larman and Vic Basili's paper in IEEE Computer, June 2003: Iterative and Incremental Development: A Brief History, and presentation slides for History and Evidence: The Evolution of Agile, Iterative vs Waterfall


• Also from Craig Larman, Iterative and Evolutionary, from Ch 7 of the book Agile and Iterative Development: a Manager's Guide


• Overcoming cultural challenges in adopting iterative development , by Clay Nelson, The Rational Edge, October 2004


• Philippe Kruchten's article From Waterfall to Iterative Development: A tough transition for project managers (Rational Edge, December 2000) and Going Over the Waterfall with the RUP (Rational Edge, April 2004)


• Per Kroll's Transitioning from Waterfall to Iterative Development (Rational Edge, April 2004)


• Joe Marasco's Iterative Development, from the book The Software Development Edge: Essays on Managing Successful Projects


• Brian G. Lyons' Managing Requirements in an Iterative Lifecycle


• Kurt Bittner's What is Iterative Development? Part III: The Management Perspective (also see Part II: the Customer Perspective , and Part I: The Developer Perspective), from the Rational Edge, Mar-May 2005


• Effective Governance practices for iterative software development, Mark Lines (Rational Edge, Feb 2005)


• Iterative Software project planning and tracking , Johanna Rothman


• Heuristics for Iterative Software Development, by Drako Sotirovski, IEEE Software May/June 2001


• Involving Customers Early and Often in a Software Development Project, Laura Rose, Rational Edge, January 2006


• Iterative Software Development - a Practical View, by Morten Korsaa et.al., from DF-16 (Datateknisk Forum), Denmark 2001 (and some very nice accompanying presentation slides)


• A Report of Development Lifecycle Methodologies Compared


• Controlling Iterative Software Development projects - The Challenge of Stakeholder & Technical Integration , by Bendik Bygstad, ICSS2004 proceedings


• Using Iterative and Incremental Development Processes in Global Software Development, from the 3rd International Workshop on Global Software Development , May 2004


• COCOMO-based effort estimation for iterative and incremental software development , Oddur Benediktsson et.al., Software Quality Journal, Nov 2003


• Tool support for iterative, adaptable, and agile software development methodologies, by Scott Henninger, 2002 ICSE Workshop on Iterative, Adaptable and Agile Processes


• Timeboxing - a process model for iterative software development , by Pankoj Jalote, from 2003 Journal on Systems and Software (also see ICICT 2005 paper on Using Defect Analysis Feedback for Improving Quality and Productivity in Iterative Software Development)

Kandt's SCM Best-Practices: Tool Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.

In this entry, I'll enumerate Kandt's Tool Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 19:

Check code in often

Kandt goes on to explain "This practice should be constrained when checking in code on the primary development branch. That is, developers should only check in working versions of code to a [primary] development branch."

The main reason given is to ensure against loss of change by making sure the code is in the repository. So although initially it might seem this practice is about committing in small talks (which I'm sure Kandt would approve), that doesn't appear to be what he's emphasizing. This really seems to be about frequent use of "Private Versions" and then, before doing a Task-Level Commit, making sure one does a Private Build and runs Unit Tests and Smoke Tests as part of the Codeline Policy.

In other words: "If it ain't checked-in, it don't exist!"

Practice 20:

Configuration management tools should provide patch utilities

The stated benefit here is to support incremental release/deployment and also remotely submitted changes over slow telecommunication connections.

Practice 21:

Do not work outside of managed workspaces

Seems like the "Private Workspace pattern to me

Practice 22:

Do not share workspaces

This is sort of reinforcing that the workspace should be private rather than shared. Having said that, while I agree shared workspaces (not include Pair-Programming) shouldn't usually be the norm, I have seen cases where it can work quite well.

In order for this to work, there must be an explicit "protocol" agreed upon up-front by all the contributors to decide who doing what, when, and to what (not just who is modifying which files when, but who is building what else and when) and/or how to rapidly communicate who is doing what when. See also work by Andre van der Hoek et.al. on Continuous Coordination: A New Paradigm for Collaborative Software Engineering Tools and related work on the "Palantir" project.

Practice 23:

When developing software on a branch other than the primary branch, regularly synchronize development with the development branch

This seems like mainlining (the Mainline pattern) and rebasing (a.k.a. Workspace Update). Only I would say that regularly synchronizing should apply to private workspaces/sandboxes even when they aren't using a private or task branch. The more general rule would seem to be the Continuous Update pattern.

Th-th-th-thats all from Mr. Kandt's list of 10 SCM principles and 23 SCM best-practices!

Kandt's SCM Best-Practices: Protection Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll enumerate Kandt's Protection Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 15:

Use a software system to perform configuration management functions

Also known as: "Use an SCM tool!"

Practice 16:

Repositories should exist on reliable physical storage elements

Practice 17:

Configuration management repositories should be periodically backed-up to non-volatile storage and purged of redundant or useless information

Practice 18:

Test and confirm the backup process

Is there really anything here to argue with? (I didn't think so :-)

Oh, and that #18 is one that is often overlooked. I can't tell you the number of times I've come across folks that think there stuff is safe "cuz it's backed-up," but their backup process didn't confirm that they were able to successfully read/extract the data that was archived.

Next up, we'll look at what Kandt identifies as Tool practices in his list of 23 SCM best-practices.

Kandt's SCM Best-Practices: Quality Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll enumerate Kandt's Quality Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 8:

All source artifacts should be under configuration control

Practice 9:

Use a Change-Control Board (CCB)

Practice 10:

Build software on a regular, preferably daily, basis, followed by invocations of regression test suites

Practice 11:

Document identified software defects

Practice 12:

Software artifacts that comprise a release should adhere to defined acceptance criteria

Practice 13:

Each software release should be regression tested before the test organization receives it

Practice 14:

Apply defect repairs to every applicable release or ongoing development effort

Most of folks in the "Agile camp" and in the "CM camp" would probably consider most of these to be "no brainers." The ones that Agilists might pick nits with are 8, 9, 11 and 14.

Control Contrarians and Wooden "Boards"

For 8 and 9, I don't think you'll find any agile folks recommending against the use of a version control system/repository. They would likely take issue with the appropriate meaning of "control." The words "control" and "change control" often set off sirens and red-flags for many agilists. And with good reason - they all too often see control attempted in a way that stifles developers and imposes highly restrictive waiting and redundancy for authorization/permission to do things; and they often see it executed as a means of preventing change instead of embracing it (see The Unchangeable Rules of Software Change).

Agilists also don't like the term CCB. Not only does the word "control" raise hackles (as already mentioned), but the term "Board" doesn't mesh very well with Agile values and perspectives on the whole concept of "team": a "board" conjures images of people isolated from the day-to-day workings and workers of the project. In reality, iteration planning meetings basically fill the CCB function for most agile projects because the iteration is usually small enough that there just isn't time to introduce new requirements within the iteration's timebox, so it's usually deferred to the next iteration.

Bugbases and Human Faces

There are many agilists who will rail against using a change/defect tracking tool. They say things like: there shouldn't be enough defects to warrant a bugbase (as opposed to a spreadsheet); rather than documenting and describing them in a database, we should just go fix 'em instead; index cards are a better for recording the bare essentials and promoting conversational dialogue and two-way face-to-face communication; tracking systems dehumanize communication and are too easily misused to the effect of hindering rather than facilitating dialogue and collaboration.

I understand all of these remarks and concerns. And while they all raise valid points, my own position is different. I consider a basic defect/issue/enhancement tracking (DIET) system to be every bit as essential as a version control tool. I still use index cards as a means of initiating dialogue and eliciting/capturing resulting needs and constraints. But then I put them into the tracking tool. Maybe if my handwriting were more legible the cards would be comprehendible enough to others, but I still think it's much more useful and convenient for tracking, organizing, sorting, searching, status accounting and generating reports (even if they just get posted to a local "information radiator").

I also think information about defects should be captured in order to identify and understand possible trends and root-causes, as well as provide evidence and history to consumers and customers (especially those requiring formal support/service-level agreements).

I do think a lot of change control implementations (and resulting tool customizations) make things harder on developers than needed, for the sake of convenience to other roles. I think that's what leaves the bitter taste in many agilists mouths and why they dislike it so much - because it holds them back and/or diverts them away from what they feel is most important: delivering value to the customer in the form of tangible, tested results. I think that's a shame because I think it doesnt' have to be that way. A few tracking tools are gaining popularity in the Agile arena, like VersionOne, and Jira.

The Monstrosity of Multiple Maintenance

Okay - I don't think that any agilists will say that a defect shouldn't be fixed in all applicable release. What they will say is that they would first do everything in their power to have one and only one mainline of development. Branching a new mainline to support a legacy release or a market/platform/project-specific variant is anathema for most agilists: it creates redundancy among code-streams (fixes have to be applied and integrated more than once) and splits the value-delivery stream into isolated streams/teams of diminished capacity.

Agilists would prefer to see a solution at a later binding time that uses design patterns of software architecture, building/releasing, licensing/distribution, & install/upgrade rather than patterns of version branching. For the most part I agree with them, but I am less wary of branching in the case of parallel development (because I know how to do it really effectively), and I am perhaps more accepting of the case of multiple releases (but I would still fight like heck against multiple variants :-)

Next up, we'll look at what Kandt identifies as Protection practices in his list of 23 SCM best-practices.

Kandt's SCM Best-Practices: Management Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll enumerate Kandt's Management Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 1:

Maintain a unique, read-only copy of each release

Also known as: create an immutable release label

Practice 2:

Control the creation, modification, and deletion of software artifacts following a defined procedure

Also known as: use version control, and agree on how you'l be doing it -- for example by identifying which SCM patterns you'll be using and their specific implementation in your project's context

Practice 3:

Create a formal approval process for requesting and approving changes

Also known as: manage change/scope, preferably by managing expectations rather than by trying to prevent change

Practice 4:

Use Change Packages

This one is more than just task-level commit, it builds on task-level-commit to provide task-based-development

Practice 5:

Use shared build processes and tools

We wrote about this in our October 2003 CM Journal article on Agile Build Management and it's March 2004 successor article on Continuous Staging

Practice 6:

A version manifest should describe each software release

This is more than just a self-identifying configuration listing of files and versions. Kandt also intends it to mean identifying the set of features, fixes, and enhancements too, as well as all open problems and issues. This is often included in the release notes for a new release. It also relates to the information necessary to satisfy a configuration audit.

Practice 7:

Segregate derived artifacts from source artifacts

Also known as: know your sources from your targets! Often times, the versioning/storage strategies used for the two may differ. (Of course, that's not the only reason to segregate them.)

Next up, we'll look at what Kandt identifies as Quality practices in his list of 23 SCM best-practices.

Kandt's SCM Principles

From the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll simply enumerate the principles from the article. In subsequent entries I'll list the 4 different sets of best-practices.

Kandt's Ten Basic Principles of SCM are:

Principle 1:

Protect critical data and other resources

Principle 2:

Monitor and control software development procedures and processes

Principle 3:

Automate processes and procedures when cost-effective

Principle 4:

Provide value to customers

Principle 5:

Software artifacts should have high-quality

Principle 6:

Software systems should be reliable

Principle 7:

Products should provide only necessary features, or those having high value

Principle 8:

Software systems should be maintainable

Principle 9:

Use critical resources efficiently

Principle 10:

Minimize development effort

Nothing particularly earth shattering here. A few interesting things to note:

• All of them are certainly well aligned with agility, or any sound engineering practices for that matter. But #7 and #10 seem especially well aligned with agility, and are often not emphasized enough in many CM circles.
• #10 in particular might surprise some folks, because I'm sure many developers might perceive CM as trying to do anything but minimize development effort, and may feel #10 is often treated as secondary and subordinate to #2
  

That's all well and good. Those things are easy to say "should" about. What's harder is to successfully do them all, and balance them all effectively when facing an SCM problem. It will be more interesting to see what Kandt's 23 SCM best-practices are, and how they manage to uphold these principles.

I'll also note that these principles seem somewhat different from the kind of SCM Principles I've been trying to compile. The things I'm looking for are less about SCM process goals, and more about SCM solution design (e.g., design principles for how to design or select an SCM best-practice that most effectively preserves the above goals). In this regard, I might consider most of the above to be goals more than principles (but with a few exceptions.)

The Unchangeable Rules of Software Change

I dont think I've ever written this down before, but I commonly say this to many a developer and development team. I often come across teams in the early stages of learning to deal with changing requirements. They typically run into two pitfalls in the following order:

Pitfall #1: No scope change-management

The very first pitfall they fall into is not having any kind of change-management for the project/requirements. They allow any and all changes in a very reactive fashion without thinking about trying to renegotiate the scope, schedule, or resources for the project/release.

Pitfall #2: Preventing scope changes

The very next time around, they run into the second pitfall: they overcompensate for being burned/bitten by the first pitfall by trying to prevent any and all changes to the scope of their next product release.

They keep insisting that the fundamental problem is they dont have completely stable detailed requirements. If the requirements were detailed enough up-front, they think their estimates would be more accurate; and if only the requirements would stop changing before a single line of code is written, then the schedule wouldn't keep fluctuating so much. It's those darn users/customers/analysts that don't know what exactly they want at the outset. It's all their fault!

The irony is that the more they try to get stable detailed requirements up-front, the more the schedule becomes protracted: first to get all the gory-details analyzed and specified up-front; and second because so many of the gory-details were either incorrect or still not detailed enough. It becomes this vicious cycle of trying to prevent change with more up-front detail, and yet things keep getting worse instead of better.

The first thing I commonly do here is explain the following:

There is a very fancy technical term that biologists use to describe completely stable systems. This highly sophisticated technical term is the word "DEAD!"

I then try to explain that we meager humans (including ourselves and our customers) are imperfect, and we have imperfect and incomplete knowledge: We don't know things, and we don't know that we don't know things, and we don't know how to find out many of those things earlier.

Then I tend to mention Phil Armour's description of the Five Orders of Ignorance and how Software is not a Product, and that software development is a therefore a knowledge creation activity which involves reducing our ignorance over time though learning and discovery about the domain (our requirements) and ourselves (our process, culture, and skill/capabilities).

At this point I then introduce them to my "tried and true, battle-proven and industry-acknowledged, Unchangeable Rules of Software Change":

Rule #0: Change is Inevitable!

The Requirements/Plans ARE going to change!

Rule #1: Resistance is Futile!

There isn’t a darn thing you can do to prevent Rule #0.

Rule #2: Change is like Quicksand -- Fighting it only makes it worse!

The more you try to deny and defy rule #1 by attempting to prevent rule #0, the worse things will get.

Rule #3: Change is like Quicksilver -- Tightening your grip makes it slip from your grasp!

The more you try to deny and defy rule #2 by attempting to precisely predict, or rigidly control change, the more erratic and onerous the result will be.

Rule #4: Embrace change to control change

The more flexible and adaptive you are at accommodating change, the more control you will have over it.

Recently I was talking to a group that was struggling with rule #2. They thought if they could only do even more detailed specification up-front (they already do a reasonable amount of up-front detail), that it would somehow eliminate problems with estimation accuracy, which in turn would alleviate problems with "conformance to plan" and prevent the details from being discovered later (because they would instead get them all "right" up-front).

Despite having plenty of historical evidence/data in this particular product to support the "inescapable truth" laid out by these rules, there still seemed to be that desire to cling to the illusion of control that we can somehow prevent such changes if only we spend more time+effort getting a more complete+perfect+detailed spec up-front.

I was searching for external validation ("not invented here") and then came across the following three things that I liked a lot:

• Phil Armour's Jan 2006 CACM article Counting Boulders and Measuring Mountains: Accuracy and the Measurement of Peaks and Projects (Phil's book The Laws of Software Process is one of my favorites)


• Chapter 2 of Karl Wieger's newest book More About Software Requirements (Microsoft Press, 2006; ISBN 0-7356-2267-1) entitled Cosmic Truths About Software Requirements"


• A recent gem of a posting from Per Kroll, Director at Rational Software and co-author of "The Rational Unified Process Made Easy", posted on the IBM/Rational "RUP" discussion forum on Jan 6, 2006:

  The fundamentally difficult hurdle managers needs to come across is that when you do not know enough to produce an accurate estimate, it will not help to do a more fine-granular plan. Rather, you need to do a list of what are the unknowns and risks preventing you from doing a more detailed estimate, and then rapidly do the analysis, design, implementation, testing, or other risk mitigation turning unknowns into knowns. This sounds as something that should not be very difficult to understand, but it really is a big mountain to climb for many people...
  
  So, where can your manager get a better understanding on these topics? Some suggestions:

  • Joe Morasco, Iterative Development,
    
  • Philippe Kruchten, From Waterfall to Iterative Development: A tough transition for project managers, The Rational Edge, December 2000.
    
  • We have a 2-day training course, Mastering Management of Iterative Development (MMID)
    
  • Craig Larman, Agile & Iterative Development: A Manager's Guide, Addison-Wesley, 2004. Excellent book providing tons of research on benefits of iterative development [see the excerpt Software Development: Iterative and Evolutionary]
    
  • Walker Royce: Software Project Management: A Unified Framework, Addison-Wesley-Longman (1998) ISBN 0-201-30958-0
    
  • Murray Cantor: Software Leadership: A Guide to Successful Software Development. Addison-Wesley Professional (2001) 208p. ISBN 0-201-70044-1

Lean Principles for Branching

A recent thread on the scrumdevelopment YahooGroup about "Scrum releases and SCM" got me thinking about a set of "Agile SCM" slides I prepared, one of which tried to apply principles of lean thinking to branching and merging for version control and their relationship to some of the SCM Patterns.

That was using an earlier version of the principles when Tom and Mary had 10 or so of them. Now they've narrowed it doesnt to seven, so I figured I'd take another stab at it:

1. Eliminate Waste – Eliminate avoidable merge-propagation (multiple maintenance), duplication (long-lived variant branches), and stale code in infrequently synchronized workspaces (partially completed work)


2. Build Quality In – Maintain codeline integrity with (preferably automated) unit & integration tests and a Codeline Policy to establish a set of invariant conditions that all checkins/commits to the codeline must preserve (e.g., running and passing all the tests :-)
   


3. Amplify Learning – Facilitate frequent feedback via frequent/continuous integration and workspace update


4. Defer Commitment (Decide as late as possible) -- Branch as late as possible! Create a label to take a "snapshot" of where you MIGHT have to branch off from, but don't actually create the branch until parallelism is needed.


5. Deliver Fast (Deliver as fast as possible) -- complete and commit change-tasks and short-lived branches (such as task-branches, private-branches, and release-prep branches) as early as possible


6. Respect People (Decide as low as possible) -- let developers reconcile merges and commit their own changes (as opposed to some "dedicated integrator/builder")


7. Optimize the "Whole" -- when/if branches are created, use the Mainline pattern to maintain a "leaner" and more manageable branching structure

Did I get it right? Did I miss anything?

Big 'A', the three pillars, and the three 'F's

In the past I've asked "What are Form and Fit for Software CM? and gotten some very interesting answers. Configuration auditing for physical, functional, and process "integrity" (correct+consistent+complete) is a commonly recurring phrase in many classical CM documents and standards. And I was curious to understand how "form, fit and function" mapped from the physical world of hardware into the virtual world of software.

I assumed the "function" part was easy to map (functionality) and that it was the other two, form and function, that were hard. I also wondered where "the three 'F's of form+fit+function originated from.

That made me wonder if it had anything to do with the three pillars of Vitrivius from classical architecture. This goes back to an earlier blog posting about Commodity, Integrity and Simplicity that also discussed the Big 'A' (Architecture) and the three 'F's.

The classical Greco-Roman architect Vitruvius described the three pillars of architecture as Utilitas, Firmitas, and Venustas: Utilitas is usually translated as utility, need, or function; Firmitas as firmness, durability, or stability of structure; and Venustas as beauty, aesthetics, or having pleasing/attractive form.

I can see how beauty or aesthetics could be translated as "form", and certainly see how "utility" could be translated as function. I'm not sure if I see a direct translation between "firmness" and "fit" (perhaps the better the "fit" the more durable the structure?)

I am wondering if form fit and function evolved on their own, separate from form, function, and durability ... or if they are related and "durability" somehow got translated into "Fit" in CM circles. What is the difference between the three pillars of architecture, and form + fit + function for configuration auditing of product integrity?