Dimensions and Views of SCM Architecture

The November issue of The Rational Edge has three articles that are closely related to my ideas about applying what we know about software & enterprise "architecture" to the domain of SCM/ALM solutions (and another article about an SCM tool vendor "eating their own dogfood"):

• UML, RUP, and the Zachman Framework: Better together, by Vitalie Temnenco
• The four dimensions of system models, by Fredrik Ferm
• A RUP-based approach to developing a data warehouse, by Scott Ambler
• Change Management at IBM Rational: How those who develop ClearQuest use the product, by Robert Pierce

[Also see some interesting articles on Requirements Engineering in the December 2006 issue of Crosstalk]

Some of you may recall my 4+2 Model Views of SCM Solution Architecture. I've since updated the picture a bit as follows (which Ive now updated over there too), click on the small image below to see a much larger one:



Anyway - reading through the above articles (particularly the one on model "dimensions" and the one on UML+RUP+Zachman together) gave me some more thoughts about my 4+2 views model, such as:

• I have "scale & diversity" as a dimension of "concerns" that bridge all views. I wonder if that corresponds to the "scale" or "level" dimension referred to the in article on dimensions of system models?


• I also have change/creation time, and decision binding time as a "dimension" of concerns. It's not explained real well though. I suppose it corresponds to considering both a "static" and "dynamic" perspective for each view. In a sense there are lifecycles (e.g., promotion cycles) for the elements in each view, as well as important times when decisions are committed to, and/or when artifacts or changes are made. When dealing with variation (diversity) across a product or system, the binding-time employed by the solution can be of great importance.


• By adding a +2 view to RUP/Kruchten's original 4+1, I end up with something that sort of maps to the 6 columns of the Zachman model. Does this mean my 4+2views are really a subset of Enterprise Architecture (EA), or might it mean that its some kind of "bridge" between the two?


• Does it really make sense for me to have Organization and data combined into a single view, or is Data really a separate view (a +3 view?), or should I not go so far as to equate metrics, queries and reports with "Data", but rather as the requirements for data (and hence the possible "bridge" mentioned above)?


• What if I compare contrast my 4+2 views with the model or views put forth by any of ITIL, RUP-SE, DODAF, TOGAF, FEAF, or others (see below)? What value does mine add that warrants being its own "thing" instead of just some poor sawed-off wannabe clone of one of the others?

Anyways, these questiosn made me want to go look-up some of these other models and views. I found some pretty good online articles for some of them (I'm sure I missed a few as well). Here they are:

• The new, improved RUP SE architecture framework
• An IBM Rational approach to the Department of Defense Architecture Framework (DoDAF) Part 1: Operational view
• An IBM Rational approach to the Department of Defense Architecture Framework (DoDAF) Part 2: Systems view
• Estimation variance and governance
• Rational Unified Process for Systems Engineering -- Part I: Introducing RUP SE Version 2.0, August 2003
• Rational Unified Process for Systems Engineering -- Part II: System Architecture, September 2003
• Rational Unified Process for Systems Engineering -- Part III: Requirements Analysis and Design, October 2003
• "How does RUP SE apply to a system of systems?"
• "Leading modern systems development"
• "Thoughts on Functional Decomposition"
• "Organizing RUP SE projects"
• Development of Federal Enterprise Architecture Framework using the Rational Unified Process and the UML
• Introducing The Open Group Architecture Framework (TOGAF), Part 1: Understand TOGAF and IT architecture in today's world
• Introducing The Open Group Architecture Framework (TOGAF), Part 2: Explore an industry standard for defining an enterprise architecture
• Introducing The Open Group Architecture Framework (TOGAF), Part 3: Create an enterprise architecture with TOGAF
• developerWorks : Architecture : New to Architecture

I welcome feedback/comments on any of my thoughts above!

Simplicity in Better Software

Back in May of this year I blogged about how Simple ain't Easy: Myths and Misunderstanding about Simplicity. That entry ended up being quite popular and very well received.

This month, a streamlined version of that entry is the featured article in the "Last Word" column for the November 2006 issue of Better Software magazine. The writing is a bit more compact, though I did end up having to trim a few thoughts I had really wanted to keep. The quotations and links from the original entry didn't make it into the article, but were instead made part of the online "Sticky Notes" for the November issue.

For those with access to the printed magazine who have a chance to read both the article and the blog-entry, I'd appreciate your feedback as to whether you like the article better, or the blog-entry better, and why.

Relating SCM Patterns to SCM Principles

Our August Agile SCM column in the CM Journal is about Relating SCM Patterns to SCM Principles. The article is pretty flimsy, barely a skeleton, and that's pretty much entirely my my fault. I meant to write more "meat" about how certain principles are the underlying forces behind several patterns. I wanted to show how the principles are strongly related, much the same way patterns in a pattern language are related. And I wanted to show how that structure shaped the relationships between the patterns as well. Rob did a really good job trying to put together what I had with what he could come up with, but I didn't really give him enough and couldn't easily convey it in a way that made it easy for him to "run with it!"

It's not that I don't see those relationships, I do. And I'm not lacking for words to describe them either. I'm having trouble describing them clearly and concisely. A big part of that is because I still don't like the names of the SCM Principles as I've described them so far. Their names currently relate to the OOD Principles they were derived from. I think that might speak to programmers, but not to folks trying to do version control (even if they also wear "developer" hats). I really want to go back and rework the names of the principles to be more simple and direct.

This is something Ive been working in my mind on and off for alkmost a decade now. To me, it is of profound important - perhaps even the most significant contribution I'll have made to the field of SCM to date. THe interest level doesn't seem to be so high on the scm-patterns list and the cmcrossroads.com forums (but that hasn't deterred me - yet :). I really do think it's not a coincidence that principles of object-oriented design also manifest themselves as fundamental principles of SCM solution design (because I think both are all about architecture - and minimizing and managing dependencies).

I'd really like feedback. On this stuff so if you've had a chance to read thru the June, July and August Agile CM columns, I'm going to create a new YahooGroup about Agile-CM for discussing these and other Agile CM issues. Hopefully this new agile-cm group will have a healthy balance of both SCM folks and Agile development folks so we can have some constructive multi-faceted discussions.

Agile SCM Principles - From OOD to TBD+CBV+POB

I finally finished a set of articles I'd been working on for almost 10 years on and off on the subject of "translating" principles of OOD into principles of SCM. See the following:

• Principles of Agile Version Control - Part I
• Principles of Agile Version Control - Part II

The principles of OOD translated into principles of Task-Based Development (TBD), Container-based Versioning (CBV), and Project-Oriented Branching (POB).

Here are the principles that I translated. Most of them are from Robert Martin's book Agile Software Development: Principles, Patterns, and Practices, but a couple of them are from The Pragmatic Programmers:

Principles of Class Design
SRP	The Single Responsibility Principle	A class should have one, and only one, reason to change.
OCP	The Open-Closed Principle	A class should be open for extension (of its behavior) but closed against modification (of its contents).
LSP	The Liskov Substitution Principle	Derived classes must be substitutable for (require no more, and ensure no less than) their base classes.
DIP	The Dependency Inversion Principle	Depend on abstract interfaces, not on concrete details.
ISP	The Interface Segregation Principle	Make fine grained interfaces that are client specific.
LOD	The Law Of Demeter (Principle of Least Assumed Knowledge)	Objects and their methods should assume as little as possible about the structure or properties of other objects (including its own subcomponents).
DRY	The "Don't Repeat Yourself" Principle	Every piece of knowledge should have one authoritative, unambiguous representation within the system.
Principles of Package Design
REP	The Release Reuse Equivalency Principle	The granule of reuse is the granule of release.
CCP	The Common Closure Principle	Classes that change together are packaged together.
CRP	The Common Reuse Principle	Classes that are used together are packaged together.
Principles of Package Coupling
ADP	The Acyclic Dependencies Principle	The dependency graph of packages shall contain no cycles.
SDP	The Stable Dependencies Principle	Depend in the direction of stability.
SAP	The Stable Abstractions Principle	Abstractness increases with stability.

Here is what I ended-up translating them into. Note that some of the principles translated into more than one principle for version control because they applied to more than one of changes/workspaces, baselines, and codelines. I'm not real thrilled about the names & acronyms for several of them and am open to alternative names & acronyms:

General Principles of Container-Based Versioning
The Content Encapsulation Principle (CEP)	All version-control knowledge should have a single authoritative, unambiguous representation within the system that is its "container. In all other contexts, the container should be referenced instead of duplicating or referencing its content.
The Container-Based Dependency Principle (CBDP)	Depend upon named containers, not upon their specific contents or context. More specifically, the contents of changes and workspaces should depend upon named configurations/codelines.
The Identification Insulation Principle (IDIP)	A unique name should not identify any parts of its context nor or of its related containers (parent, child or sibling) that are subject to evolutionary change.
The Acyclic Dependencies Principle (ADP)	The dependency graph of changes, configurations, and codelines should have no cycles.
Principles of Task-Based Development
The Single-Threaded Workspace Principle (STWP)	A private workspace should be used for one and only one development change at a time.
The Change Identification Principle (CHIP)	A change should clearly correspond to one, and only one, development task.
The Change Auditability Principle (CHAP)	A change should be made auditably visible within its resulting configuration.
The Change/Task Transaction Principle (CHTP)	The granule of work is the transaction of change.
Principles of Baseline Management
The Baseline Integrity Principle (BLIP)	A baseline's historical integrity must be preserved - it must always accurately correspond to what its content was at the time it was baselined.
The Promotion Leveling Principle (PLP)	Define fine-grained promotion-levels that are consumer/role-specific.
The Integration/Promotion Principle (IPP)	The scope of promotion is the unit of integration & baselining
Principles of Codeline Management
The Serial Commit Principle (SCP)	A codeline, or workspace, should receive changes (commits/updates) to a component from only one source at a time.
The Codeline Flow Principle (CLFP)	A codeline's flow of value must be maintained - it should be open for evolution, but closed against disruption of the progress/collaboration of its users.
The Codeline Integrity Principle (CLIP)	Newly committed versions of a codeline should consistently be no less correct or complete than the previous version of the codeline.
The Collaboration/Flow Integration Principle (CFLIP)	The throughput of collaboration is the cumulative flow of integrated changes.
The Incremental Integration Principle (IIP)	Define frequent integration milestones that are client-valued.
Principles of Branching & Merging
The Codeline Nesting Principle (CLNP)	Child codelines should merge and converge back to (and be shorter-lived than) their base/parent codeline.
The Progressive-Synchronization Principle (PSP)	Synchronizing change should flow in the direction of historical progress (from past to present, or from present to future): more conservative codelines should not sync-up with more progressive codelines; more progressive codelines should sync-up with more conservative codelines.
The Codeline Branching Principle (CLBP)	Create child branches for value-streams that cannot "go with the flow" of the parent.
The Stable Promotion Principle (SPP)	Changes and configurations should be promoted in the direction of increasing stability.
The Stable History Principle (SHIP)	A codeline should be as stable as it is "historical": The less evolved it is (and hence more mature/conservative), the more stable it must be.

You can read the 2nd article to see which version-control principles were derived from which OOD principles. Like I mentioned before, I'm not real thrilled about the names & acronyms for several of them and am open to alternative names & acronyms. So please share your feedback on that (or on any of the principles, and how they were "derived").

Agile SCM Principles - Design Smells

I finally finished a set of articles I'd been working on for almost 10 years on and off on the subject of "translating" principles of OOD into principles of SCM. See the following:

• Principles of Agile Version Control - Part I
• Principles of Agile Version Control - Part II

In an August 2005 blog-entry on SCM Design smells, I tried to translate the design smells that Robert Martin. wrote up in his book on Agile principles, Patterns and Practices. In the first of the two articles above, I think I was much more successful at translating them into the version-control domain. Here is what I came up with ...

Symptoms of Poor Version Control

Rigidity/Inertia:

The software is difficult to integrate and deploy/upgrade because every update impacts, or is impacted by, dependencies upon other parts of the development, integration, or deployment environment

Fragility/Inconsistency:

Builds are easily “broken” because integrating new changes impacts other fixes/features/enhancements that are seemingly unrelated to the change that was just made, or changes keep disappearing/reappearing or are difficult to identify/reproduce

Immobility/Inactivity:

New fixes/features/enhancements take too long to develop because configurations and their updates take a long time to integrate/propagate or build & test.

Viscosity/Friction:

The “friction” of software process against the development flow of client-valued changes is too high because the process has an inappropriate degree of ceremony or control

Needless Complexity/Tedium:

Procedures and tasks are overly onerous and/or error-prone because of too many procedural roles/steps/artifacts, too fine-grained “micro” tracking/status-accounting, overly strict enforcement or rigid and inflexible workflow

Needless Repetition/Redundancy:

The version-control process exhibits excessive branching/merging, workspaces or baselining in order to copy the contents of files, changes and versions to maintain multiple views of the codebase.

Opacity/Incomprehensibility:

It is difficult to understand and disentangle the branching/merging and baselining schemes into a simple and transparent flow of changes for multiple tasks and features developed from multiple collaborating teams working on multiple projects from multiple locations for multiple customers

In my next entry I'll start describing the actual principles and their translations.

Trustworthy Transparency over Tiresome Traceability

If there was an Agile CM Manifesto, then this statement should be part of it!

Trustworthy Transparency over Tiresome Traceability

Note that my position on traceability is more "lean" than "agile" I suspect. I base this on the XP & Scrum centric views that were expressed in the March 2004 YahooGroup discussion thread Why Traceability? Can it be Agile? I think the "tests over traceability" is probably a valid summary of the XP/Scrum perspective from that thread.

I think myself and David Anderson would probably say something more along the lines of "transparency over traceability", where we acknowledge the important goals that traceability is trying to fulfill (I'm not sure the XP community embraces all of those that I identified as the "8 reasons" and "6 facets" that I identified in my paper on traceability dissected). David in particular has written in the past about "trustworthy transparency" and "naked projects" (projects that are so transparent and visible in their status/accounting that they seem "naked").

I also differ strongly with the many vocal opinions expressed in XP community when it comes to the use of tools for tracking requests/changes: I'm strongly in favor of using a "good" tracking tool. I think index cards are a great and valuable "tool" for eliciting dialogue and interaction with the "customer" (and I use them for this purpose, along with post-it notes). But I believe index cards simply do not "cut it" as a serious means of storing, tracking, sorting, searching, slicing & dicing development/change requests).

I do believe an extent of traceability is necessary, and that it's not necessarily "agile", but that it can be, and should be, "lean" and streamlined, and should serve the purpose of transparency, visibility and status-accounting rather than being a goal in itself. And I think there are several strategies and tactics that can be employed to achieve "lean" traceability in service to "trustworthy transparency and friction-free metrics."

I think that a "lean" approach of traceability would focus on the following:

1. Flow: If one uses "single piece flow" and does changes at the granularity that TDD mandates, then software-level requirements, design, coding, and testing are all part of the same task, and tracking them to a single record-id in the change-tracking system and version-control tool would actually go a long ways toward traceability (its much more work & intermediate artifacts when those activities are all separated over time (different lifecycle phases), space (different artifacts) and people (different roles/organizations). When traceability efforts noticeably interfere with "flow" is when agilists will start screaming.


2. Minimizing intermediate artifacts and other perceived forms of "waste" (overspecifying requirements or too much requirements "up front") because fewer artifacts means fewer things to trace.
   


3. Collocating both people & artifacts (the former for communication, the latter for "locality of reference") for those artifacts that are deemed necessary.


4. Coarse-Granularity and Modularity/Factoring of what is traced: tracing at the highest practical level of granularity (e.g., is it practical to trace to the individual requirement or the use-case? To the line of code, or to the method/subroutine, or to the class/module) - this would be about "simple design" and "(re)factoring)" as it applies to the structure of the traced entities and their relationships.


5. Transparent, frictionless automation of the terribly taxing and tiresome tedium of traceability: focus on taking the tedium out of manual traceability and have it streamlined and automated as much as possible, ideally happening seamlessly behind the seems (like with Jane Huang's event-based traceability (EBT), or thru the use of a common environment "event" catcher within Eclipse or MS Team System server), probably using a task-based, test-driven (TDD), or feature-driven (FDD) approach.

Many of these concepts and more are embodied in Sam Guckenheimer's recent book on Software Engineering with Microsoft Visual Studio Team System. I found this book to be surprisingly good (outstanding even), and not at all what I was expecting given the apparent tool/vendor-specific nature suggested by the title. The value-up paradigm and most of the other concepts and values in the book are very well aligned with agility while still meeting the needs of more rigorous ceremony in their software and systems engineering efforts.

I'll close with a description of a recent presentation by David Anderson on Changing the Software Engineering Culture with Trustworthy Transparency:

"Modern software tooling innovation allows the tracking of work performed by engineers and transparent reporting of that work in various formats suitable for everything from day-to-day management and team organization to monthly and quarterly senior executive reporting. Modern work item tracking is coupled to version control systems and aware of analysis, design, coding and testing transitions. This makes it not only transparent but trustworthy. Not only can a tool tell you the health of a project based on the state of completion of every work item, but this information is reliable and trustworthy because it is tightly coupled to the system of software engineering and the artifacts produced by it.

The age of trustworthy transparency in software engineering is upon us. Trustworthy transparency changes the culture in an organization and enables change that unleashes significant gains in productivity and initial quality. However, transparency and managing based on objective study of reality strains existing software engineering culture as all the old rules, obfuscation, economies of truth, wishful thinking and subjective decision making must be cast aside. What can you expect, how will you cope and how can you harness the power of trustworthy transparency in your organization?"

As someone with a strong Unix and Open-Source heritage, I've long regarded Microsoft as "the evil empire" and loathed their operating system and browser and ALM tools. But in the last 3 years or so they've acquired a number of people in the Agile and ALM community that I highly respect (Brian White, Sam Guckenheimer, David Anderson, Ward Cunningham, James Newkirk) and the products these folks have worked on look incredibly impressive to me (even tho not all of them are still with Microsoft), plus I'm quite impressed with the whole of their Software Factories vision and approach ...

I actually may have to start liking them (or at least part of them :-). Don't get me wrong! I'm still a big fan Unix (and Mac OS/X), Open-Source, and more recently Eclipse, ALF and Corona; But the competing stuff from the folks in Redmond is looking exceedingly more and more impressive to me. Working on those kinds of things with those people would be an incredible experience I think (now if only I could do that without having to relocate from Chicago or spend 25% or more of my time traveling ;-).

Pragmatic Multi-Variant Management

I had a rather lengthy post on this subject on the Pragmatic Programming Yahoo group ...

My advice as a CM guy is you do NOT want to use branches or branching to solve this problem. If you have to manage/support multiple "configurations" of functionality for multiple markets/customers, branching is just about the worst way to do it (and this is coming from someone who is an advocate of branching, when done the right way for the right reasons).

The issue is likely to be one of "Binding Time". In your case, you would prefer the binding-time to be either at build-time (such as conditional compilation or linking), or release-engineering time (packaging up the "right" sets of files, including configuration files, for distributing to a customer), install/upgrade-time, or run-time.

One of the prefered ways to do this is with Business-Rules; particularly when it comes to decisions of "variable" policies and/or functionality. With GUI's depending on the kind of variation, you might resort to either conditional compilation or else simply creating separate source-files (instead of branching the same file).

There were some substantial discussions on this topic on CMCrossroads forums that yielded many practical insights and further resources. I recommend them:

• CM for product variations
• Branching or using #ifdef?

fantamango77 wrote:

I can say I tried out different strategies in earlier projects already. But not with such a big project. And none of the ways I took made me happy. It always ended up in a kind of hell: Configuration hell, branching hell, inheritance hell.

Yes - and some of those "circles" of h*ll are far worse than others when you look at the overall costs and effort. The bottom-line is that if you have to support multiple "variants" rather than a single evolving "line", you are adding complexity, and there is no way you are going to be able to sweep it under the rug or make it go away.

So it all comes down to which techniques and strategies in which "dimensions" of the project/product will prove most effective at minimizing and managing dependencies, effort and impact by most effectively allowing you to leverage proven principles such as encapsulation, cohesion, modularity, abstraction, etc. in the most efficient ways.

Think about what aspects or dimensions of your project/product are required to be "variable" in order for you to support multiple variants. Do your "variants" need to differ ...

• in functional/behavioral dimensions
• along organizational boundaries
• along multiple platforms/environments
• along temporal (evolution) dimensions
• along project dimensions

Figure out which one or two of these are the primary "dimensions" of variability you need to support. Then find the set of mechanisms and techniques that apply to it.

For example, if you need to allow variability primarily along functional and environmental "dimensions", then which of those "dimensions" does version branching operate within? Version branching operates primarily within the space of evolution/time (concurrent, parallel, and distributed development).

So temporally-based techniques are not going to be best suited to handling variation in the behavioral or environmental dimensions, as the isolation/insulation they provide does not most effectively minimize, localize or encapsulate the kinds of dependencies in the non-time-based aspects of the system.

Differences in policy and/or mechanism are typically best handled using a business-rules approach to deliver a single codebase with multiple possible configurations of rules and rule-settings.

Differences in platforms are best handled by well known design and architecture patterns like Wrapper-Facade, and numerous patterns from the Gang-of-Four design patterns book.

Differences in behavior may be best handled by functional/feature selection and deselection "design patterns" like Configurator, to enable or disable features and/or services at post-development binding-times.

Inheritance may be useful in some cases, if the type of configuration needed really does fit a single hierarchical model of increasing specialization. In other cases, an aspect-oriented approach might be better.

Also think about the following in terms of what needs to "vary" and what needs to stay the same:

• Interface -vs- Implementation -vs- Integration
• Container -vs- Content -vs- Context

This kind of commonality and variability analysis helps isolate the fundamental dimensions or aspects of variation that need to apply to your project. If something needs to be able to vary while something else doesnt, then "encapsulate the thing that varies" using techniques that separate interface from implementation (or implementation from integration, or etc.) in ways that "keep it [structure] shy, DRY, and tell the other guy."

You might end up using a combination of strategies depending on the different aspects of variation you require and the "dimension" in which each one operates.

Unchangeable Rules of Software Change - Redux

I put together a couple of my earlier blog-entries on the topic of software change and iterative development and developed them into an article in the February 2006 issue of CMCrossroads Journal. The article is entitled The Unchangeable Rules of Software Change (just like the earlier blog-entry) and updates some of what I had blogged about earlier.

In addition to the first three commonly recurring pitfalls encountered when first faced with the reality of these "unchangeable rules", I identified three additional pitfalls that typically occur when first attempting iterative development. It also has a few more iterative development resources than my previous follow-up blog-entry. Lastly, I expanded the rule-set by one, adding the "quicksilver" rule to the "quicksand" rule as noted below:

The Unchangeable Rules of Software Change

Rule #0: Change is Inevitable!

The Requirements/Plans ARE going to change!

Rule #1: Resistance is Futile!

There isn't a darn thing you can do to prevent Rule #0.

Rule #2: Change is like Quicksand -- Fighting it only makes it worse!

The more you try to deny and defy rule #1 by attempting to prevent rule #0, the worse things will get.

Rule #3: Change is like Quicksilver -- Tightening your grip makes it slip from your grasp!

The more you try to deny and defy rule #2 by attempting to precisely predict, or rigidly control change, the more erratic and onerous the result will be.

Rule #4: Embrace change to control change

The more flexible and adaptive you are at accommodating change, the more control you will have over your outcomes

You can read the whole article here!

Kandt's SCM Best-Practices: Tool Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.

In this entry, I'll enumerate Kandt's Tool Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 19:

Check code in often

Kandt goes on to explain "This practice should be constrained when checking in code on the primary development branch. That is, developers should only check in working versions of code to a [primary] development branch."

The main reason given is to ensure against loss of change by making sure the code is in the repository. So although initially it might seem this practice is about committing in small talks (which I'm sure Kandt would approve), that doesn't appear to be what he's emphasizing. This really seems to be about frequent use of "Private Versions" and then, before doing a Task-Level Commit, making sure one does a Private Build and runs Unit Tests and Smoke Tests as part of the Codeline Policy.

In other words: "If it ain't checked-in, it don't exist!"

Practice 20:

Configuration management tools should provide patch utilities

The stated benefit here is to support incremental release/deployment and also remotely submitted changes over slow telecommunication connections.

Practice 21:

Do not work outside of managed workspaces

Seems like the "Private Workspace pattern to me

Practice 22:

Do not share workspaces

This is sort of reinforcing that the workspace should be private rather than shared. Having said that, while I agree shared workspaces (not include Pair-Programming) shouldn't usually be the norm, I have seen cases where it can work quite well.

In order for this to work, there must be an explicit "protocol" agreed upon up-front by all the contributors to decide who doing what, when, and to what (not just who is modifying which files when, but who is building what else and when) and/or how to rapidly communicate who is doing what when. See also work by Andre van der Hoek et.al. on Continuous Coordination: A New Paradigm for Collaborative Software Engineering Tools and related work on the "Palantir" project.

Practice 23:

When developing software on a branch other than the primary branch, regularly synchronize development with the development branch

This seems like mainlining (the Mainline pattern) and rebasing (a.k.a. Workspace Update). Only I would say that regularly synchronizing should apply to private workspaces/sandboxes even when they aren't using a private or task branch. The more general rule would seem to be the Continuous Update pattern.

Th-th-th-thats all from Mr. Kandt's list of 10 SCM principles and 23 SCM best-practices!

Kandt's SCM Best-Practices: Protection Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll enumerate Kandt's Protection Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 15:

Use a software system to perform configuration management functions

Also known as: "Use an SCM tool!"

Practice 16:

Repositories should exist on reliable physical storage elements

Practice 17:

Configuration management repositories should be periodically backed-up to non-volatile storage and purged of redundant or useless information

Practice 18:

Test and confirm the backup process

Is there really anything here to argue with? (I didn't think so :-)

Oh, and that #18 is one that is often overlooked. I can't tell you the number of times I've come across folks that think there stuff is safe "cuz it's backed-up," but their backup process didn't confirm that they were able to successfully read/extract the data that was archived.

Next up, we'll look at what Kandt identifies as Tool practices in his list of 23 SCM best-practices.

Kandt's SCM Best-Practices: Quality Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll enumerate Kandt's Quality Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 8:

All source artifacts should be under configuration control

Practice 9:

Use a Change-Control Board (CCB)

Practice 10:

Build software on a regular, preferably daily, basis, followed by invocations of regression test suites

Practice 11:

Document identified software defects

Practice 12:

Software artifacts that comprise a release should adhere to defined acceptance criteria

Practice 13:

Each software release should be regression tested before the test organization receives it

Practice 14:

Apply defect repairs to every applicable release or ongoing development effort

Most of folks in the "Agile camp" and in the "CM camp" would probably consider most of these to be "no brainers." The ones that Agilists might pick nits with are 8, 9, 11 and 14.

Control Contrarians and Wooden "Boards"

For 8 and 9, I don't think you'll find any agile folks recommending against the use of a version control system/repository. They would likely take issue with the appropriate meaning of "control." The words "control" and "change control" often set off sirens and red-flags for many agilists. And with good reason - they all too often see control attempted in a way that stifles developers and imposes highly restrictive waiting and redundancy for authorization/permission to do things; and they often see it executed as a means of preventing change instead of embracing it (see The Unchangeable Rules of Software Change).

Agilists also don't like the term CCB. Not only does the word "control" raise hackles (as already mentioned), but the term "Board" doesn't mesh very well with Agile values and perspectives on the whole concept of "team": a "board" conjures images of people isolated from the day-to-day workings and workers of the project. In reality, iteration planning meetings basically fill the CCB function for most agile projects because the iteration is usually small enough that there just isn't time to introduce new requirements within the iteration's timebox, so it's usually deferred to the next iteration.

Bugbases and Human Faces

There are many agilists who will rail against using a change/defect tracking tool. They say things like: there shouldn't be enough defects to warrant a bugbase (as opposed to a spreadsheet); rather than documenting and describing them in a database, we should just go fix 'em instead; index cards are a better for recording the bare essentials and promoting conversational dialogue and two-way face-to-face communication; tracking systems dehumanize communication and are too easily misused to the effect of hindering rather than facilitating dialogue and collaboration.

I understand all of these remarks and concerns. And while they all raise valid points, my own position is different. I consider a basic defect/issue/enhancement tracking (DIET) system to be every bit as essential as a version control tool. I still use index cards as a means of initiating dialogue and eliciting/capturing resulting needs and constraints. But then I put them into the tracking tool. Maybe if my handwriting were more legible the cards would be comprehendible enough to others, but I still think it's much more useful and convenient for tracking, organizing, sorting, searching, status accounting and generating reports (even if they just get posted to a local "information radiator").

I also think information about defects should be captured in order to identify and understand possible trends and root-causes, as well as provide evidence and history to consumers and customers (especially those requiring formal support/service-level agreements).

I do think a lot of change control implementations (and resulting tool customizations) make things harder on developers than needed, for the sake of convenience to other roles. I think that's what leaves the bitter taste in many agilists mouths and why they dislike it so much - because it holds them back and/or diverts them away from what they feel is most important: delivering value to the customer in the form of tangible, tested results. I think that's a shame because I think it doesnt' have to be that way. A few tracking tools are gaining popularity in the Agile arena, like VersionOne, and Jira.

The Monstrosity of Multiple Maintenance

Okay - I don't think that any agilists will say that a defect shouldn't be fixed in all applicable release. What they will say is that they would first do everything in their power to have one and only one mainline of development. Branching a new mainline to support a legacy release or a market/platform/project-specific variant is anathema for most agilists: it creates redundancy among code-streams (fixes have to be applied and integrated more than once) and splits the value-delivery stream into isolated streams/teams of diminished capacity.

Agilists would prefer to see a solution at a later binding time that uses design patterns of software architecture, building/releasing, licensing/distribution, & install/upgrade rather than patterns of version branching. For the most part I agree with them, but I am less wary of branching in the case of parallel development (because I know how to do it really effectively), and I am perhaps more accepting of the case of multiple releases (but I would still fight like heck against multiple variants :-)

Next up, we'll look at what Kandt identifies as Protection practices in his list of 23 SCM best-practices.

Kandt's SCM Best-Practices: Management Practices

More from the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll enumerate Kandt's Management Best-Practices for SCM (in priority order, with the most important ones listed first):

Practice 1:

Maintain a unique, read-only copy of each release

Also known as: create an immutable release label

Practice 2:

Control the creation, modification, and deletion of software artifacts following a defined procedure

Also known as: use version control, and agree on how you'l be doing it -- for example by identifying which SCM patterns you'll be using and their specific implementation in your project's context

Practice 3:

Create a formal approval process for requesting and approving changes

Also known as: manage change/scope, preferably by managing expectations rather than by trying to prevent change

Practice 4:

Use Change Packages

This one is more than just task-level commit, it builds on task-level-commit to provide task-based-development

Practice 5:

Use shared build processes and tools

We wrote about this in our October 2003 CM Journal article on Agile Build Management and it's March 2004 successor article on Continuous Staging

Practice 6:

A version manifest should describe each software release

This is more than just a self-identifying configuration listing of files and versions. Kandt also intends it to mean identifying the set of features, fixes, and enhancements too, as well as all open problems and issues. This is often included in the release notes for a new release. It also relates to the information necessary to satisfy a configuration audit.

Practice 7:

Segregate derived artifacts from source artifacts

Also known as: know your sources from your targets! Often times, the versioning/storage strategies used for the two may differ. (Of course, that's not the only reason to segregate them.)

Next up, we'll look at what Kandt identifies as Quality practices in his list of 23 SCM best-practices.

Kandt's SCM Principles

From the paper Software Configuration Management Principles and Best Practices, by Ronald Kirk Kandt, appearing in the Proceedings of PROFES2002, the 4th International Conference on Product-Focused Software Process Improvement, Rovanieme Finland, December 2002.

In this article, Ronald Kandt describes ten basic principles that support configuration management activities, and then goes on to describe twenty three "fundamental" practices, which are split across the categories of: Management Practices, Quality Practices, Protection Practices, and Tool Practices.


In this entry, I'll simply enumerate the principles from the article. In subsequent entries I'll list the 4 different sets of best-practices.

Kandt's Ten Basic Principles of SCM are:

Principle 1:

Protect critical data and other resources

Principle 2:

Monitor and control software development procedures and processes

Principle 3:

Automate processes and procedures when cost-effective

Principle 4:

Provide value to customers

Principle 5:

Software artifacts should have high-quality

Principle 6:

Software systems should be reliable

Principle 7:

Products should provide only necessary features, or those having high value

Principle 8:

Software systems should be maintainable

Principle 9:

Use critical resources efficiently

Principle 10:

Minimize development effort

Nothing particularly earth shattering here. A few interesting things to note:

• All of them are certainly well aligned with agility, or any sound engineering practices for that matter. But #7 and #10 seem especially well aligned with agility, and are often not emphasized enough in many CM circles.
• #10 in particular might surprise some folks, because I'm sure many developers might perceive CM as trying to do anything but minimize development effort, and may feel #10 is often treated as secondary and subordinate to #2
  

That's all well and good. Those things are easy to say "should" about. What's harder is to successfully do them all, and balance them all effectively when facing an SCM problem. It will be more interesting to see what Kandt's 23 SCM best-practices are, and how they manage to uphold these principles.

I'll also note that these principles seem somewhat different from the kind of SCM Principles I've been trying to compile. The things I'm looking for are less about SCM process goals, and more about SCM solution design (e.g., design principles for how to design or select an SCM best-practice that most effectively preserves the above goals). In this regard, I might consider most of the above to be goals more than principles (but with a few exceptions.)

The Unchangeable Rules of Software Change

I dont think I've ever written this down before, but I commonly say this to many a developer and development team. I often come across teams in the early stages of learning to deal with changing requirements. They typically run into two pitfalls in the following order:

Pitfall #1: No scope change-management

The very first pitfall they fall into is not having any kind of change-management for the project/requirements. They allow any and all changes in a very reactive fashion without thinking about trying to renegotiate the scope, schedule, or resources for the project/release.

Pitfall #2: Preventing scope changes

The very next time around, they run into the second pitfall: they overcompensate for being burned/bitten by the first pitfall by trying to prevent any and all changes to the scope of their next product release.

They keep insisting that the fundamental problem is they dont have completely stable detailed requirements. If the requirements were detailed enough up-front, they think their estimates would be more accurate; and if only the requirements would stop changing before a single line of code is written, then the schedule wouldn't keep fluctuating so much. It's those darn users/customers/analysts that don't know what exactly they want at the outset. It's all their fault!

The irony is that the more they try to get stable detailed requirements up-front, the more the schedule becomes protracted: first to get all the gory-details analyzed and specified up-front; and second because so many of the gory-details were either incorrect or still not detailed enough. It becomes this vicious cycle of trying to prevent change with more up-front detail, and yet things keep getting worse instead of better.

The first thing I commonly do here is explain the following:

There is a very fancy technical term that biologists use to describe completely stable systems. This highly sophisticated technical term is the word "DEAD!"

I then try to explain that we meager humans (including ourselves and our customers) are imperfect, and we have imperfect and incomplete knowledge: We don't know things, and we don't know that we don't know things, and we don't know how to find out many of those things earlier.

Then I tend to mention Phil Armour's description of the Five Orders of Ignorance and how Software is not a Product, and that software development is a therefore a knowledge creation activity which involves reducing our ignorance over time though learning and discovery about the domain (our requirements) and ourselves (our process, culture, and skill/capabilities).

At this point I then introduce them to my "tried and true, battle-proven and industry-acknowledged, Unchangeable Rules of Software Change":

Rule #0: Change is Inevitable!

The Requirements/Plans ARE going to change!

Rule #1: Resistance is Futile!

There isn’t a darn thing you can do to prevent Rule #0.

Rule #2: Change is like Quicksand -- Fighting it only makes it worse!

The more you try to deny and defy rule #1 by attempting to prevent rule #0, the worse things will get.

Rule #3: Change is like Quicksilver -- Tightening your grip makes it slip from your grasp!

The more you try to deny and defy rule #2 by attempting to precisely predict, or rigidly control change, the more erratic and onerous the result will be.

Rule #4: Embrace change to control change

The more flexible and adaptive you are at accommodating change, the more control you will have over it.

Recently I was talking to a group that was struggling with rule #2. They thought if they could only do even more detailed specification up-front (they already do a reasonable amount of up-front detail), that it would somehow eliminate problems with estimation accuracy, which in turn would alleviate problems with "conformance to plan" and prevent the details from being discovered later (because they would instead get them all "right" up-front).

Despite having plenty of historical evidence/data in this particular product to support the "inescapable truth" laid out by these rules, there still seemed to be that desire to cling to the illusion of control that we can somehow prevent such changes if only we spend more time+effort getting a more complete+perfect+detailed spec up-front.

I was searching for external validation ("not invented here") and then came across the following three things that I liked a lot:

• Phil Armour's Jan 2006 CACM article Counting Boulders and Measuring Mountains: Accuracy and the Measurement of Peaks and Projects (Phil's book The Laws of Software Process is one of my favorites)


• Chapter 2 of Karl Wieger's newest book More About Software Requirements (Microsoft Press, 2006; ISBN 0-7356-2267-1) entitled Cosmic Truths About Software Requirements"


• A recent gem of a posting from Per Kroll, Director at Rational Software and co-author of "The Rational Unified Process Made Easy", posted on the IBM/Rational "RUP" discussion forum on Jan 6, 2006:

  The fundamentally difficult hurdle managers needs to come across is that when you do not know enough to produce an accurate estimate, it will not help to do a more fine-granular plan. Rather, you need to do a list of what are the unknowns and risks preventing you from doing a more detailed estimate, and then rapidly do the analysis, design, implementation, testing, or other risk mitigation turning unknowns into knowns. This sounds as something that should not be very difficult to understand, but it really is a big mountain to climb for many people...
  
  So, where can your manager get a better understanding on these topics? Some suggestions:

  • Joe Morasco, Iterative Development,
    
  • Philippe Kruchten, From Waterfall to Iterative Development: A tough transition for project managers, The Rational Edge, December 2000.
    
  • We have a 2-day training course, Mastering Management of Iterative Development (MMID)
    
  • Craig Larman, Agile & Iterative Development: A Manager's Guide, Addison-Wesley, 2004. Excellent book providing tons of research on benefits of iterative development [see the excerpt Software Development: Iterative and Evolutionary]
    
  • Walker Royce: Software Project Management: A Unified Framework, Addison-Wesley-Longman (1998) ISBN 0-201-30958-0
    
  • Murray Cantor: Software Leadership: A Guide to Successful Software Development. Addison-Wesley Professional (2001) 208p. ISBN 0-201-70044-1

Lean Principles for Branching

A recent thread on the scrumdevelopment YahooGroup about "Scrum releases and SCM" got me thinking about a set of "Agile SCM" slides I prepared, one of which tried to apply principles of lean thinking to branching and merging for version control and their relationship to some of the SCM Patterns.

That was using an earlier version of the principles when Tom and Mary had 10 or so of them. Now they've narrowed it doesnt to seven, so I figured I'd take another stab at it:

1. Eliminate Waste – Eliminate avoidable merge-propagation (multiple maintenance), duplication (long-lived variant branches), and stale code in infrequently synchronized workspaces (partially completed work)


2. Build Quality In – Maintain codeline integrity with (preferably automated) unit & integration tests and a Codeline Policy to establish a set of invariant conditions that all checkins/commits to the codeline must preserve (e.g., running and passing all the tests :-)
   


3. Amplify Learning – Facilitate frequent feedback via frequent/continuous integration and workspace update


4. Defer Commitment (Decide as late as possible) -- Branch as late as possible! Create a label to take a "snapshot" of where you MIGHT have to branch off from, but don't actually create the branch until parallelism is needed.


5. Deliver Fast (Deliver as fast as possible) -- complete and commit change-tasks and short-lived branches (such as task-branches, private-branches, and release-prep branches) as early as possible


6. Respect People (Decide as low as possible) -- let developers reconcile merges and commit their own changes (as opposed to some "dedicated integrator/builder")


7. Optimize the "Whole" -- when/if branches are created, use the Mainline pattern to maintain a "leaner" and more manageable branching structure

Did I get it right? Did I miss anything?

Big 'A', the three pillars, and the three 'F's

In the past I've asked "What are Form and Fit for Software CM? and gotten some very interesting answers. Configuration auditing for physical, functional, and process "integrity" (correct+consistent+complete) is a commonly recurring phrase in many classical CM documents and standards. And I was curious to understand how "form, fit and function" mapped from the physical world of hardware into the virtual world of software.

I assumed the "function" part was easy to map (functionality) and that it was the other two, form and function, that were hard. I also wondered where "the three 'F's of form+fit+function originated from.

That made me wonder if it had anything to do with the three pillars of Vitrivius from classical architecture. This goes back to an earlier blog posting about Commodity, Integrity and Simplicity that also discussed the Big 'A' (Architecture) and the three 'F's.

The classical Greco-Roman architect Vitruvius described the three pillars of architecture as Utilitas, Firmitas, and Venustas: Utilitas is usually translated as utility, need, or function; Firmitas as firmness, durability, or stability of structure; and Venustas as beauty, aesthetics, or having pleasing/attractive form.

I can see how beauty or aesthetics could be translated as "form", and certainly see how "utility" could be translated as function. I'm not sure if I see a direct translation between "firmness" and "fit" (perhaps the better the "fit" the more durable the structure?)

I am wondering if form fit and function evolved on their own, separate from form, function, and durability ... or if they are related and "durability" somehow got translated into "Fit" in CM circles. What is the difference between the three pillars of architecture, and form + fit + function for configuration auditing of product integrity?

4+2 Views of SCM Principles?

In my last blog-entry I wondered if the interface segregation principle (ISP) translated into something about baselines/configuration, or codelines, or workspaces, or build-management. Then I asked if it might possibly relate to all them,

Here's a somewhat scary thought (or "cool" depending on your perspective), what if the majority of Robert Martin's (Uncle Bob's) Principles of OOD each have a sensible, but different "translation" for each of the architectural views in my 4+2 Views Model of SCM/ALM Solution Architecture? (See the figure below for a quick visual refresher.)




Thus far, the SCM principles I've "mapped" from the object-oriented domain revolve around baselines and configurations, tho I did have one foray into codeline packaging. What if each "view" defined a handful of object-types that we want to minimize and manage dependencies for? And what if those principles manifested themselves differently in each of the different SCM/ALM subdomains of:

• change control (project-view)
  
• version control (evolution view)
  
• artifact (requirements, models, code, tests, docs) hierarchy and build management (product view)
  
• workspace/repository/site management and application integration & synchronization (environment view)
  
• workflow and process design (process view)
• teaming, inter-group coordination and interfaces/expectations (organization view)

What might the principles translate into in each of those views, and how would the interplay between those principles give rise to the patterns already captured today regarding recurring best-practices for the use of baselines, codelines, workspaces, repositories, sites, change requests & tasks, etc.

Interface Segregation and Configuration Promotion

I've been thinking more about the Interface Segregation Principle (abbreviated as "ISP") from (Uncle) Bob Martin's Principles of Object-Oriented Design.

The "short version" of ISP in the initial article states that:
=> "Clients should NOT be forced to depend on interfaces that they do not use."

The summary of ISP in Uncle Bob's website says it differently:
=> "Make fine grained interfaces that are client specific."

In previous blog-entries, I've wondered how this might correctly translate into an SCM principle (if at all).

• In Change-Packaging Principles, I wondered if maybe it corresponds to change-segregation or incremental integration: Make fine-grained incremental changes that are behavior-specific. (i.e., partition your task into separately verifiable/testable yet minimal increments of behavior.)


• On the scm-patterns list I wondered if maybe it corresponds to composite baselines: composing baselines of other, more fine-grained baselines


• Now I'm thinking maybe it corresponds to promotion lifecycle modeling and defining the promotion-levels in a promotion-lifecycle of a configuration-item (e.g., a build).

Why am I thinking this?

I guess I'm trying to go back to the basis of my means of comparison: configurations (and hence baselines) as "objects." If a configuration is an object, then what is an interface of a configuration, and what is a fine-grained interface (or "service")?

If I am thinking in terms of configuration building, then the interface for building the object (configuration) is the equivalent of Make/ANT "methods" and targets for a given item: (e.g., standard make targets like "clean", "all", "doc", "dist", and certain standard conventions for makeflags). That is certainly a plausible translation.

But if I am thinking in terms of baselining and supporting CM-mandated needs for things like reproducibility, repeatability, traceability, from the perspective of the folks who "consume" the baseline (it's clients), then maybe the different consumers of a baseline need different interfaces.

If those consumers end up each "consuming" the baseline at different times in the development lifecycle (e.g., coding, building, testing, etc.) then perhaps that defines what the promotion model and promotion levels should be for that configuration.

What if they aren't at different times in the lifecycle? What if they are at the same time?

Then I guess it matters if the different consumers are interested in the same elements of the baseline. If they're not, maybe that identifies a need for composite baseline.

What if they aren't at different times and aren't for different elements, but rather the same sets of elements?

Then maybe that identifies different purposes (and services) needed by different consumers for the same configuration at the same time. Building -versus- Coding might be one such example. Would branching -versus- labeling be another? (i.e. "services" provided by a configuration as represented by a "label" as opposed to by a "codeline", or a "workspace"?)

What if no one of these is the "right" interpretation? What if it's ALL of them?

Then that would be very interesting indeed. If the result encompassed the interfaces/services provided by different Promotion-Levels, Make/ANT-targets, branch -vs- label -vs- workspace, then I don't even know what I would call such a principle. I might have to call it something like the Configuration ISP, or the Representation separation principle, or the manifestation segregation principle, or ....

What, if anything, do YOU think the ISP might mean when applied to Software CM and software configurations as represented by a build/label/codeline/workspace?

Commodity, Integrity, Simplicity

In a previous blog-entry on the subject of perishable -vs- durable value, I wrote about how business value is depreciable and therefore the business value of a feature is a perishable commodity. I then went on to describe what I thought were more durable forms of value: Integrity and Simplicity.

• I defined Integrity as a triplet of related properties: {Correctness, Consistency, Completeness}. Integrity is a property of a deliverable item such as a feature, a configuration or a configuration item. So a feature or item has "integrity" if it is correct, consistent and complete.


• I also defined Simplicity as a triplet of related properties: {Clarity, Cohesiveness (Coherency), Conciseness}. So a feature, item, or logical entity is "simple" if it is clear, cohesive and concise.

I then asked the question:

What about "form, fit and function"? Are "form" and "fit" also components of perishable value?

What I've been thinking since then is that the perishable form of value is the extrinsic value that it is given by the customer. From the end-consumers perspective, what they perceive as the form, fit, and function of the deliverable is what makes it valuable or not. We might call type of value "Commodity" or "Marketability". [Note: There are several things I both like and dislike about both those possible names, so please comment if you have a preference for one over the other (or for something else) and let me know why.]

I suggested this in a posting to the continuousintegration YahooGroup entitled "Commodity, Integrity, Simplicity (was Re: Extreme Frequency and Extreme Integrity)". Some relevant excerpts from the discussion:

Commodity is customer-desired Form, Fit and Function. ... Commodity has to do with what requirements are most valued by the customer at a given time. I think maybe those requirements are in terms of "Form, Fit, and Function". Which requirements those are and how much they are valued is most definitely time-sensitive. When I add "commodity"-based value to a codebase, I am adding time-sensitive perishable value that can depreciate or greatly fluctuate over time.
...

[from Ron Jeffries]:

A thing, to me, has integrity and simplicity but is a commodity.

I thought about this. And I completely agree - that probably is the main thing that makes the word "commodity" stand-out apart from the other two like "one of those things that just doesnt belong" with them.

Then I think about it some more, and I think, maybe the thing that makes it seem so "wrong" when listed with the other two is perhaps what is so "right" about it after all. Maybe it's a good think to think that a feature (or "story") is a commodity.

Maybe that's what it is first and foremost (a commodity) that we should always keep in mind, and where the most direct value to the customer is perceived. And maybe those other two things (integrity and simplicity) are the "secret sauce" that make all the difference in how we do it:

• Maybe the integrity is the "first derivative" that gives us velocity AND continuity at a sustainable pace.


• And maybe when we throw in simplicity, that is the second derivative of value, and it maybe harder for the customer to see directly, but when we do it right, that gives us more than just continuity+sustainability, it also gives us the acceleration to adaptiveness and responsiveness and "agility" to overcome that cost-of-change curve.

...

[follow-up from Ron Jeffries]:

However, a bit further insight (or what I use in place of insight) for why it troubles me. A "commodity" is a kind of product with value, but it is a fungible one. A commodity is a product usually sold in bulk at a price per item or per carload. One potato is like every other potato. A story/feature, in an important sense, isn't like every other story/feature.

Thanks Ron for all the thoughtful feedback. You are spot-on of course. And that notion of a commodity as a bulk shipment or mass purchase of units definitely "kills" the notion of value I'm trying to get at.

I'm still at a loss for a word/term that I like better. Marketability perhaps? It's more syllables than I'd like, although there is a precedent set for it in the book Software By Numbers in its use of an "Incremental Funding Method" (IFM) with "Minimal, Marketable Features" (MMFs).

So to my readers that have read this far ... what is your take on all of this talk about commodity/marketability and "perishable value"? Are commodity, integrity, and simplicity each just different perspectives of form, fit, and function, where:

• "commodity/marketability" would be the customer view
  
• "integrity" would be the view of requirements analysts/engineers, V&V/QA, and CM
• "simplicity" would be the view of the developers and architects

What do form, fit and function mean for software anyway?

• Is it container, context and content?
• Is it interface, integration and implementation?

And how should that all trace back to our discussion about value and whether that value is extrinsic or intrinsic, and whether it is perishable, durable, or latent/emergent?

I admit I dont have a lot of coherent thoughts here, just a lot of incoherent ramblings and inconsistent questions. Let me know how you think this should all make sense (or if it shouldn't).

When to Commit: Perishable Value and Durable Value

We had a recent (and interesting) discussion on the scm-patterns YahooGroup about the notion of "value" and Frank Schophuizen got me thinking about what is the "value" associated with a configuration or a codeline: how does value increase or decrease when a configuration is "promoted" or when/if the codeline is branched/split?

Agile methods often talk about business value. They work on features in order of the most business-value. They eschew activities and artifacts that don't directly contribute to delivery business value. etc...

David Anderson, in several of his articles and blogs at agilemanagement.net, notes that the value of a feature (or other "piece" of functionality) is not dependent upon the cost to produce it, but upon what a customer is willing to pay for it. Therefore the value of a feature is perishable and depreciates over time:

• The longer it takes to receive delivery of a feature, the less a customer may begin to value it.


• If it doesn't get shipped in the appropriate market-window of opportunity, the value may be significantly lost.


• If the lead-time to market for the feature is too long, then competitive advantage may be lost and your competitor may be able to offer it to them sooner than you can, resulting in possible price competition, loss of sale or business
  

So business value is depreciable; and the value of a feature is a perishable commodity.

Might there be certain aspects to business value that are not perishable? Might there be certain aspects that are of durable value? Is it only the functionality associated with the feature that is of perishable value? Might the associated "quality" be of more durable value?

I've seen the argument arise in Agile/XP forums about whether or not one should "commit" one's changes every time the code passes the tests, or if one should wait until after refactoring, or even until more functionality is implemented (to make it "worth" the time/effort to update/rebase, reconcile merge conflicts and then commit).

Granted, I can always use the Private Versions pattern to checkin my changes at any time (certainly any time they are correct+consistent) without also committing them to the codeline for the rest of the team to see and use. So, assuming that the issue is not merely having it secured in the repository (private versions), when is it appropriate to commit my changes to the codeline for the rest of the team to (re)use?

If refactoring is a "behavior preserving transformation" of the structure of the code, and if it improves the design and makes it "simpler", then is "good design" or "simplicity" something that adds durable value to the implementation of a running, tested feature? Kent Beck's initial criteria for "simple code" (and how to know when you are done refactoring your latest change) was described in an XPMagazine article by Ron Jeffries as the following, in order of importance:

1. it passes all the tests (correctly :-)
   


2. it contains no redundancy (the DRY principle: Don't Repeat Yourself)
   


3. it expresses every thought we intended it to convey about the program (i.e. reveals all our intent, and intends all that it reveals)


4. it minimizes the size and number of classes and methods

If I squint a little when I read thru the above, it almost looks like it's saying the same thing that writing-instructors and editor's say about good writing! It should be: correct, consistent, complete, clear and concise!

I have often heard "correct, consistent and complete" used as a definition of product integrity. So maybe integrity is an aspect of durable value! And I have sometimes heard simplicity defined as "clear and concise" or "clear, concise and coherent/cohesive" (where "concise" would be interpreted as having very ruthlessly rooted out all unnecessary/extraneous or repeated verbage and thoughts). So maybe simplicity is another aspect of durable value.

And maybe integrity is not enough, and simplicity is needed too! That could possibly explain why it might make more sense to wait until after a small change has been refactored (simplified) before committing it instead of waiting only until it is correct+consistent+complete.

Perhaps the question "when should I commit my changes?" might be answered by saying "whenever I can assure that I am adding more value than I might otherwise be subtracting by introducing a change into a 'stable' configuration/codeline!"

• If my functionality isn't even working, then it's subtracting a lot of value, even if did get it into the customer's hands sooner. It causes problems (and costs) for my organization and team to fix it, has less value to the customer if it doesn't work, and can damage the trust I've built (or am attempting to build) in my relationship with that customer


• if my functionality is working, but the code isn't sufficiently simple, the resulting lack of clarity, presence of redundancy or unnecessary dependency can make it a lot harder (and more costly) for my teammates to add their changes on top of mine


• if I wait too long, and/or don't decompose my features into small enough working, testable increments of change, then the business value of the functionality I am waiting to commit is depreciating!

Now I just have to figure out some easy and objective means of figuring out the "amount" of value I have added or subtracted :-)

So are "integrity" (correct + consistent + complete) and "simplicity" (clear + concise + coherent/cohesive) components of durable value? Is functionality the only form of perishable value?

What about "form, fit and function"? Are "form" and "fit" also components of perishable value? Am I onto something or just spinning around in circles?