Brad Appleton's ACME Blog

SOA, Mashups, Mashed Knees and Surgery

2009-08-20T09:23:00.000-05:00

Today is my birthday - I had arthroscopic knee surgery last night and am feeling pretty good so far (happy birthday to me). I know I still have a lot of meds/painkillers in my system and that its going to feel more uncomfortable the next few days. I'm still hobbling around on crutches but I'm feeling much more confident that I can attend (and present at) Agile2009.

I received some very good books on Mashups and SOA a few months back and I'm finally getting a chance to look at them in some more detail. They really are quite good! I think Mashups are the "latest frontier" for realizing the promise of SOA, and a natural evolution from Wiki-webs. Here are the books:

SOA Design Patterns, by Thomas Erl
Mashup Patterns: Designs and Examples for the Modern Enterprise, by Michael Ogrinz
Mashups: Strategies for the Modern Enterprise, by J. Jeffrey Hanson

If you follow the links above to the InformIT.com homepage for each of the above books, you'll find some good excerpts and related articles that will give a preview of what to expect so you can judge for yourself.

BOOK: Running an Agile Project

2009-08-13T00:12:00.001-05:00

First, on a personal note, I had the misfortune to tear cartilage in my right knee a couple days ago and will require surgery to repair/remove it. I'm hobbling around on crutches for the time being. I hope I can still attend (and present at) Agile2009.

Anyway, my review of Mike Holcombe's book Running an Agile Software Development Project appears in this month's Agile Journal.

Running an Agile Software Development Project is an interesting book. On the surface it looks like it would be very academic, because the author, Mike Holcombe, was a University Professor at the time, and running an “agile software development factory” of students (albeit for a real commercial development shop). And yet what is described in the contents is very much the practical, real-world results of running agile projects with those same people for real IT software development work.

[...]

Overall, I found Running an Agile Software Development Project to be interesting and enjoyable. It still seems just a bit academic for my taste, and probably wouldn’t be the first book I would recommend on the subject unless it was for a classroom audience (in which case this book would be an excellent one to use).

You can read the full review here!

WANTED: Seeking Single Agile Knowledge Development Tool-set

2009-08-07T00:21:00.005-05:00

I'll be presenting at Agile2009 in Chicago on the Tools for Agility stage on Tuesday 25 August, 4:45pm-5:30pm.

Here is my session description from http://agile2009.org/node/2762

WANTED: Seeking Single Agile Knowledge Development Tool-set

Aren’t code, backlog-items, tests, designs & documents all just different forms of system knowledge at different levels of detail? Why can’t the same tools help refactor, browse, search, and provide build/test automation for non-code forms of knowledge without requiring a separate tool/repository for each format? This talk is intended as a challenge to tool vendors/developers to see how this simple treatment of all non-code items as part of a single, unified project knowledge-base can be at once both immensely powerful, and imminently practical, without requiring too much added complexity.

Process/Mechanics

Approximately ~30 minutes of slides/presentation to “make the case” for why this approach is useful and desirable, followed by discussion of challenges (to and from tool developers/vendors, as well as the rest of the audience) as to its usefulness and benefits, and why their current tools can’t easily do so and why the should or should not be easy to enhancement them to do it.

Outline

Software development as knowledge development

Source-Code as knowledge
Requirements (Stories) and Tests as knowledge
Other usable forms of project knowledge (e.g., build scripts & configuration, build/test result reports version control history/comments, online help & other supporting docs)

How would I do this?

Refactoring Knowledge (thinking about the rest of the “knowledge” the way we think about the code, and its habitability, navigability, and structure/refactoring)
Applying other agile-practices (like TDD, Continuous Integration, etc.) to non-code knowledge development
Wiki-based skins, DSLs, and use-cases/design as domain-driven Wiki-word definitions
- Patterns (giving things names), Retrospectives results and lessons learned
Viewing, searching EVERYTHING (even the code) via wiki?
The “Wu-Wei” of Traceability (Tracing without Tracing)
Versioning and operating on wiki-like entities, just like with code (e.g., making “working copies”, branches and tags)

DISCUSSION & CHALLENGE: Why can’t (or how can) YOUR tools do this!

Begin audience discussion/dialogue: Why can’t a tool like Eclipse or a Wiki-based CMS (such as Confluence) be used as a front-end to browsing/refactoring and navigating ALL the knowledge of the system? (not just code, but stories, tests, build/config data, CI reports, backlog, retrospective lessons).

What makes it easy/hard for these and other tools like any of the following to do this, or to be simple “skins” or plug-ins on top of only a handful of tools instead of a whole kitchen sink full of separate tools and repositories.

Eclipse, Confluence, Twiki?
FIT, Fitnesse, Contour
Doxygen / Javadoc
Trac / Agile-Trac?
Jira / Remedy

See also a blog-entry of mine entitled Can I have just one repository please?

Learning outcomes

Thinking about Agile development as knowledge management & creation
Current barriers to using agile techniques and supporting tools for non-code artifacts
What do refactoring, TDD, continuous integration, etc. mean for non-code artifacts (and why you should care)
Use of wikis for organizing, structuring and refactoring ALL system knowledge
Why manual traceability is unnecessary (and even comes for free) when using such an approach

Primary target persona

Tomas Tool-smith/Tool-vendor

Other target personas

(download PDF here)

Studies on Effectiveness of TDD?

2009-08-01T12:06:00.003-05:00

This question came-up in a discussion earlier this week: Do we know of published studies on this subject? A quick Google-search turned up the following for me ...

George Dinwiddie's page on Studies of Test-Driven Development has links to a dozen or so, with summaries
A March 2009 InfoQ.com article Empirical Studies Show Test Driven Development Improves Quality about a paper published in the Journal of Empirical Software Engineering
An XP2009 paper on Measuring the impact of testing on code structure in Test Driven Development: metrics and empirical analysis.
A Survey of Evidence for TDD in Academia (2008), published in ACM SIGCSE Bulletin
Empirical Evidence of Agile Methods (2007) has a nice slide or two devoted to TDD
A Univ. of Karlsruhe study on The Effect of Experience on the TDD process (2007)
An ITEA 2006 paper on TDD Empirical Body of Evidence
A 2006 Univ. of Kansas Ph.d thesis on An empirical evaluation of the impact of test-driven development on software quality
An ISESE 2006 paper on Evaluating Advantages of Test Driven Development: a Controlled Experiment with Professionals
VTT Case Study of TDD in Mobile Software Software Development (2005)

Any others? Any comments on the above?

Resources on Retrospectives

2009-07-26T00:55:00.001-05:00

I found a really good resource-list from George Dinwiddie on Introspection and Retrospectives that includes the following list of resources (mostly patterns & techniques) about conducting retrospectives. It contains many (but not all) of the links below:

RestrospectiveTechniques compiled by George Dinwiddie
Agile Retrospectives Resource Wiki
Singing the Songs of Project Experience: Patterns and Retrospectives by Linda Rising and Esther Derby from Cutter IT Journal (PDF)
Restrospective Agility by Tim Mackinnon in Objective View (PDF)
http://www.retrospectives.com/ Norm Kerth's website.
Esther Derby's
- Seven Reasons to Have a Retrospective
- 6 Reasons *not* to have a Retrospective
- What a Retrospective Ain't
- Two more ways to gather data
- Five Tips for Retrospective Leaders and Meeting Moderators and Tips for Retrospective Facilitators
Rachel Davies'
- Heartbeat Retrospectives
- How To: Live and Learn with Retrospectives
- Refactoring Your Development Process with Retrospectives
- Marco Abis' Retrospectives IN and ON action
Sample chapters 3 & 5 from Agile Retrospectives: Making Good Teams Great by Esther Derby and Diana Larsen:
- Chapter 3, Leading Retrospectives
- Chapter 5, Selected Activities
- Chapter 10, Make it So
- a 50min talk/presentation from Esther & Diana on YouTube
Ellen Gottesdiener's Team Retrospectives — for better iterative assessment looks at the topic from a RUP perspective.
Bill Wake's Some Patterns for Iteration Retrospectives
James Carr's Retrospective Patterns
Ian Burgess' Agile Retrospectives Lessons Learned
Ilja Preuss' Retrospective Anti-Patterns
Article: Leveraging Agile Retrospectives to Evolve Your Architecture
Simon Baker on Retrospective Using Appreciative Inquiry (see similar article by Diana Larsen)
Ben Coombs' A Retrospective based on Agile Values
Mark Levison's Agile Games for Making Retrospectives Interesting
Kris Blake's Retrospective Agenda for High-Performing Teams
Scrum Alliance - Seven Ways to Revitalize Your Sprint Retrospectives
Fabio Periera's Goal-Driven Retrospective
Mishkin Berteig's Agile Work: Cheat-Sheet for Retrospectives
Agile Journal - Iteration and Release Retrospectives: The Natural Rhythm for Agile Measurement
Retrospectives Content on InfoQ.com/retrospectives, including ...
- InfoQ: Heartbeat Retrospectives to Amplify Team
- InfoQ: Making Retrospective Changes Stick
- InfoQ: How Long Should Retrospectives Last?
- InfoQ: First (Forgotten?) Rule Of The Retrospective: Follow Through
- InfoQ: Annotated Burn-Down Charts Help During Retrospectives
- InfoQ: Frequent Retrospectives Accelerate Learning and Improvement
Jim Shore's brief recap and retrospective format from The Art of Agile Development (0596527675)

Refactoring @ Scale

2009-07-17T01:35:00.007-05:00

In my previous post, Refactoring for Agility, I posted an outline and some thoughts for Part I of an Overview on Refactoring. Now I'm ready to post on Part II which is about refactoring @ scale. By "at scale" I mean in the larger context of other agile practices, as well as for large projects.

PART II - REFACTORING @ SCALE

1. Scaling-Up

To scale refactoring for larger projects, some additional techniques & issues must be added to the mix.
Note that this is “in addition to” (not “instead of”)

Refactoring In-the-Small	Refactoring @ Scale
Small, Fast & Frequent Refactorings	Larger, Periodic & Planned Restructurings
Emergent Design	Incremental Design & Evolutionary Architecture
Deferred Refactoring	Restructuring & Technical Debt
Code Smells	Architecture Smells
Design Principles & Patterns	Software Modifiability Tactics
Simple/Clean Code	Supple/Domain-Driven Design

2. Emergent Design

Emergent Design is a fancy name for the resulting design that “emerges” from the synergy of combining Refactoring together with TDD, Continuous Integration and Automated Testing.
Emergent Design means that ...

3. Technical Debt [a.k.a. Design Debt]
4. Restructuring Technical Debt

If we accrue a non-trivial amount of technical debt, we can’t simply “refactor” it away.
Paying it off typically requires restructuring efforts (or even reengineering) that must be planned.
Iteration plans must accommodate specific tasks for these restructuring efforts (or even be dedicated to restructuring).
Ignoring it, or deferring it for very long is not a viable option!

5. Overview of Restructuring

Identifies higher-level issues (“architecture smells”) that typically represent violations of known principles of good software architecture & design.
Periodically applies larger-scale “refactorings” and/or many small refactorings that were previously deferred.
The goal is to “pay down technical debt” in order to limit the increasing costs of accumulated complexity.
Typically requires a concerted effort that must be separately planned.
Uses not only design patterns/principles, but also architectural patterns/principles, as well as software modifiability tactics.

6. Refactoring vs. Restructuring

See Martin Fowler's description in Refactoring Malapropism

7. Restructure Periodically

Restructuring is often associated with absent or neglectful refactoring and/or design.
But … Any large software project spanning multiple teams eventually needs restructuring.
- Even in the presence expert-level architecture, design & continuous refactoring
- This is just a reality of software evolution/entropy
Therefore … Large software projects should assume that periodic restructuring will be necessary, and should plan accordingly to:
- Clean-up accumulated code-smells and apply numerous refactorings that were deferred but are now sorely needed,
- Address architecture smells by applying restructurings, patterns and modifiability tactics that have broader impact.

8. Architecture Smells
See Stefan Roock's and Martin Lippert's book Refactoring in Large Software Projects (There was an earlier version of their work entitled Large Refactorings").

Smells in Dependency Graphs
Smells in Inheritance Hierarchies
Smells in Packages
Smells in Subsystems
Smells in Layers

9. Software Modifiability Tactics

Localize Changes (increase cohesion)
Prevent Ripple Effects (reduce coupling)
Defer Binding-time (defer decision-making)

10. When to Consider/Plan Restructuring?
11. Evolutionary Architecture

Software Architecture concerns infrastructure elements that must exist before you can begin execution.
- Architecture is about things that are hard to change later, it is difficult to allow an architecture to emerge.
- For large projects, this includes high-level organization of the system into functionality/elements that will be allocated to separate teams.
Key techniques of Evolutionary Architecture include:
- Deferring Irreversible Decisions to the “Last Responsible Moment” (LRM Principle)
- Architectural “Spike”, Architecture Iteration and/or Spike Iteration

12. Incremental Design

Design evolves incrementally, iteration by iteration, based on current business priorities and discovered technical limitations.
Incremental Design …
- Does not prohibit thinking about higher-level design.
- Does encourage planning in detail only what will be constructed soon.
Focus is on “Just Enough, Just-In-Time” :
- Specifying too much detail too soon causes more rework later.
- But doing less now and saving the rest for later should not require significantly more work later than it would today.
We must do Just Enough Design Initially to attain the right balance of anticipation and adaptation.

13. Just Enough Design Initially (JEDI)

Initial design (before coding) is still necessary.
- This use of “JEDI” was coined by Stephen Palmer as part of Feature-Driven Development (FDD)
Basic rule of thumb to tell when “JEDI” is achieved:
- At iteration-scope, when, after one pass through the iteration backlog, modeling in small groups does not produce any new classes or associations of real significance.
- At task/TDD scope, when we have defined enough structure & interface(s) to know specifically what code to write/test, precisely where to write it, and exactly how to invoke it.
Techniques of “the JEDI way” include:
- Collaborative Domain Modeling and Color Modeling [from FDD]
- Supple Design techniques & patterns
- Domain-Driven Design (a.k.a. DDD -- see domaindrivendesign.org)
- Design Blitz & other Agile Modeling techniques (see agilemodeling.com)

14. Supple Design & DDD

Domain-Driven Design (DDD) approaches modeling the core logic of the software by focusing on the domain.
- The basic idea is the design should directly reflect the core business domain and domain-logic of the problem to solve
- This helps understanding the problem as well as the implementation and increases maintainability of the software.
DDD uses common principles and patterns as "building blocks" to model & create a "supple design”
- Supple: pliant, malleable, limber, yielding or changing readily.
- The design is firm yet flexible, with structure and intent both clearly conveyed and deeply realized by the code.
Patterns of Supple Design include: Intention-Revealing Interfaces, Ubiquitous Language, Side-Effect-Free Functions, Assertions, Conceptual Contours, Standalone Classes, Closure of Operations, Declarative Style

15. Resources:

Resources on Emergent Design and Evolutionary Architecture

Neal Ford, Evolutionary Architecture and Emergent Design series of articles on IBM developerWorks:
Emergent Design: The Evolutionary Nature of Professional Software Development, by Scott L. Bain
Architecture in an Agile Organization by Chris Sterling (online)

Resources on Restructuring

InfoQ: Michael Stal on Architecture Refactoring
Refactoring in Large Software Projects: Performing Complex Restructurings Successfully, by Martin Lippert, Stephen Roock (also an earlier version online)
Object-oriented Reengineering Patterns, by S. Demeyer, S. Ducasse & O. Nierstrasz (freely downloadable online)

Resources on Technical Debt

Technical Debt and Design Death:How to ensure you can deliver business value in the future as well as in the present; by Kane Mar and Michael James
Design debt economics: A vocabulary for describing the causes, costs, and cures for software maintainability problems;by John Elm, IBM developerWorks, June 2009
Managing Technical Debt and 10X Software Development - Technical Debt, by Steve McConnell
Managing Technical Debt; by Tom Brazier
Repaying Technical Debt; by Simon Baker
Software Debt: Depreciation of Value for Software Assets and Technical Debt: Reducing Internal Quality for Short-term Gain, by Chris Sterling
Software Entropy: Don’t Tolerate Broken Windows, and Zero-Tolerance Construction by Andy Hunt and Dave Thomas

Resources on Modifiability Tactics

Modifiability Tactics, SEI @ CMU Technical Report CMU/SEI-2007-TR-002
Modifiability Tactics, Chapter 5 section 3 of Software Architecture in Practice
Understanding Architectural Patterns in Terms of Tactics and Models
Design and Tactics, Lecture notes from an Aalborg University course on Database and Software Architecture

Resources on Supple Design & DDD

Wikipedia page on DDD
DDD Intro Article
DDD – a Brief Introduction
DDD: Supple Design Patterns
DDD Pattern Summaries
Free mini-eBook: Domain-Driven Design Quickly
Yet another free mini-eBook: Step-by-Step Guide to DDD
See also domaindrivendesign.org and infoq.com/domain-driven-design

Refactoring for Agility

2009-07-16T01:32:00.003-05:00

Some of you might have guessed from my recent posts on Emergent Design, Technical Debt, JEDI Programming, and 5S Qualities of Well Designed, Well-Factored Code, that I've been looking into trying to teach the fundamentals of refactoring and how it scales to larger projects. I've gathered some references and quotes and some ideas for slides that I wanted to bounce around on my blog.

Here is an outline and some thoughts for part I of some slides ...

PART I - REFACTORING FOR AGILITY

1. Overview of Refactoring

Identifies design-maintenance issues (“code smells”) that typically represent violations of known principles of good design.
Incrementally and iteratively applies a set of design improvement techniques (“refactorings”).
The goal is to minimize complexity & duplication in order to maximize simplicity & ease-of-change.
Encourages the “right” design details to emerge “just-in-time” with minimal guesswork/rework.
Scaling-up includes the use of periodic restructuring, initial & incremental design (“just enough”), and evolutionary architecture.

2. Refactoring Defined [cite definition(s)]

3. Refactoring Myths -- Refactoring is NOT …

“Rework” – redesigning things that could, and should, have been designed correctly in the first place.
Design “gold-plating” – work that adds no business value, and merely serves to stroke the egos of perfectionists who are out of touch with business reality.
Miscellaneous code “tidying” – the kind that is “nice to have,” but should only happen when the team has some slack-time, and is a luxury we can do without, without any serious consequences.
A license to “hack” – avoiding any and all initial design & analysis and instead jumping straight to coding with no “real” design.
Reengineering – large-scale restructuring that requires a concerted effort over the course of several weeks/months to re-write or re-architect significant parts of the system.

4. Refactoring IS …

A systematic approach to source-code “hygiene” that minimizes the chances of introducing bugs
Improving the design of the code after it has been written
A behavior-preserving transformation of source-code structure
The process of simplifying & consolidating a work-product by making several, small, successive revisions focused on: preserving correctness, removing redundancy, revealing thoughts & intentions, and improving clarity & conciseness.
A disciplined way of making changes while exposing the project to significantly less risk.
An effective means to address the economic reality of software growth/complexity by reducing & amortizing its cost throughout the daily business of code development & maintenance activities.

5. Why Refactor?
6. How to Refactor
7. Rules of Clean Code
8. Rules for Simple Code
9. The Steps of Refactoring
10. Code Smells
11. Categories of Refactorings

Small Refactorings
Larger Refactorings/Restructurings
Each category contains as many as a dozen or more refactorings, most of which are catalogued at http://refactoring.com/catalog/

12. Refactorings (Some refactorings from real projects)

See http://refactoring.com/catalog/ for an up-to-date list (and the “Refactoring to Patterns” catalog too)

13. What to do if …?

I spot a “smell” that is not already known or catalogued?
There is no specific known/catalogued “refactoring” for what I think I need?

14. When to Refactor?

While adding functionality
While fixing a bug
While reviewing code
After coding the same/similar thing for the third time (to “factor out” the duplication)
A.k.a.: The Rule of Three: 3 strikes and you refactor.
After the third time you deferred refactoring a change, for any reason [The Rule of Three, again]
Before the end of the iteration if you haven’t been following The Rule of Three

15. Refactor Continually
16. When NOT to Refactor?

When the build is broken or tests don’t pass
When it would compromise meeting an impending deadline or commitment
When the code in question really just needs to be re-written “from scratch”
When it would modify code/interfaces that could significantly impact/break other work (e.g.: Published/public interfaces and protocols, Database schemas/tables/operations)
Sometimes we must defer refactoring for later and/or plan for subsequent restructuring

17. Refactoring to Patterns & Principles
Software Design Principles and Design Patterns are the underlying foundation for Refactoring:

Code smells (a.k.a “code pathologies”): Signal a possible violation of design principles, Suggest which refactoring may be needed
Refactoring: Correct a design principle violation (at least partially), Converge toward common design patterns
Design Patterns: Reconcile forces among conflicting design concerns,Restore balance between competing design principles
Design Principles: Lead us to attain desired design qualities/attributes

18. Design Attributes/Code Qualities
Qualities of Highly Maintainable Software:

Loose Coupling & High Cohesion
Hierarchy (Structural Decomposition)
Abstraction, Encapsulation & Modularity
Sufficiency, Parsimony and Primitiveness
Readability
Testability
Modifiability
Serviceability

19. Design Principles: SOLID, SoC, DRY, Shy

The SOLID Principles of Object-Oriented Design (from Uncle Bob)
The SoC Principle: Separation of Concerns — separate interface from implementation, policy from mechanism, behavior from construction, commands from queries, ...
The DRY Principle: Don’t Repeat Yourself (Eliminate Duplication), Single Point of Truth
The “Structure-Shy” Principle: (“Tell, Don’t Ask!”), The Law of Demeter, Principle of Least Assumed Knowledge

20. Other Acronyms of Simple/Agile Design

OAOO – Say Things Once And Only Once (restatement of the DRY principle)
DTSTTCPW – Do The Simplest Thing That Could Possibly Work! (restatement of the KISS principle)
YAGNI – You Aren’t Gonna Need It!
The LRM Principle: Defer Commitment of Irreversible Decisions to the Last Responsible Moment!
BDUF – Big Design Up-Front! (vs. JEDI)
JEDI – Just Enough Design Initially/In-front!
DDD – Domain-Driven Design

21. Design Patterns
22. Summary: Refactoring for Agility

Successively applies small behavior-preserving transformations to eliminate code smells
Based on proven design principles and patterns for achieving maintainability & modifiability
Good automated testing is a prerequisite
Refactoring is not rewriting, rework or restructuring
With refactoring, we continuously invest nominal effort to reduce the risk & cycle-time of changes
The goal is to minimize complexity & duplication in order to maximize simplicity & ease-of-change.
Practiced in a highly disciplined manner, it promotes:
- Sufficient functionality
- Simple & clean code
- Supple design
- Serviceable software
- Sustainable team velocity

23. Resources:
Code Smells:

Introduction to Code Smells and Refactoring from TestingEducation.org
Code Smells and Refactoring to Patterns, by Josh Kerievsky
Refactoring and Code Smells
The original WikiWiki CodeSmells page
Martin Fowler's Code Smells page
A Catalogue of Bad Smells in Code, by Martin Fowler and Kent Beck
A Taxonomy of Bad Code Smells
CodingHorror.com's list of Code Smells
Smells to Refactorings Quick Reference, by Josh Kerievsky
Gene Garcia's mapping of CodeSmells to Refactorings
Using Static Analysis Tools to Identify Code Smells, IBM developerWorks article by Paul Duvall

Design Principles:

S.O.L.I.D. Principles e-Book (free) http://www.lostechies.com/content/pablo_ebook.aspx
An O-O Primer, http://www.rgoarchitects.com/Files/ooprimer.pdf
The Principles of Object-Oriented Design: http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod
Design Principles and Patterns, http://www.objectmentor.com/resources/articles/Principles_and_Patterns.pdf and http://www.oodesign.com/
The Art of SoC (Separation of Concerns) http://www.ctrl-shift-b.com/2008/01/art-of-separation-of-concerns.html
Tell, Don’t Ask, http://www.pragprog.com/articles/tell-dont-ask
O-O in One Sentence: Keep it Shy, DRY, and Tell the Other Guy! http://media.pragprog.com/articles/may_04_oo1.pdf

Design Patterns:
- Online Resources:

Wikipedia on Design Patterns
The Patterns Home page: hillside.net/patterns
Design Patterns Reference and Patterns Tutorials
Patterns & Software: Essential Concepts & Terminology

- Books:

Design Patterns, by the “Gang of Four” (the book that started it all)
POSA Series of books on Pattern-Oriented Software Architecture
Head First Design Patterns, from the O’Reilly Head First Series
Refactoring to Patterns; by Joshua Kerievsky
Agile Software Development, Principles, Patterns, and Practices by Robert C. Martin
Implementation Patterns; by Kent Beck
Design Patterns Explained (2ed); by Alan Shalloway, James Trott

Other Agile Design Slogans:

XP Simplicity Rules, http://c2.com/cgi/wiki?XpSimplicityRules
Essential XP: Simple Design, http://www.xprogramming.com/xpmag/expEmergentDesign.htm
DTSTTCPW - http://c2.com/cgi/wiki?DoTheSimplestThingThatCouldPossiblyWork
OAOO - http://c2.com/cgi/wiki?OnceAndOnlyOnce
YAGNI - http://en.wikipedia.org/wiki/You_Ain%27t_Gonna_Need_It
LRM - http://stevesmithblog.com/blog/delaying-decisions, http://vig.pearsoned.co.uk/samplechapter/0321150783.pdf
BDUF - http://en.wikipedia.org/wiki/Big_Design_Up_Front
JEDI - http://www.featuredrivendevelopment.com/node/507
DDD - http://www.domaindrivendesign.org/

Mercurial, Git and Scala

2009-07-14T01:02:00.001-05:00

Three more brand new books I just received that are worth mentioning ...

Version Control with Git: Powerful tools and techniques for collaborative software development, by Jon Loeliger, O'Reilly, 2009

Mercurial: The Definitive Guide -- Modern Software for Collaboration, by Bryan O'Sullivan, O'Reilly, 2009 (also see the online version at hgbook.red-bean.com)

Programming Scala: Tackle Multi-Core Complexity on the Java Virtual Machine,
by Venkat Subramaniam, Pragmatic Programmers, 2009

For those who may not know ...

Mercurial (like Git) is an open-source, distributed version-control system.

While Scala is the "hot" new JVM-based programming language that not only contains the "best of both worlds" from other statically-typed languages (like Java) as well as dynamic languages like Groovy, but also the best of both worlds from object-oriented programming languages (OOPLs) and functional programming languages (like Erlang and Haskell). Plus it has message-passing and actor-based support for concurrency (like Erlang) while still being strongly typed and having access to all your favorite Java APIs (and anything else that compiles to JVM). Not too mention that Scala has a kick-butt Web development framework called Lift that is a next-generation framework to the likes of Rails/Grails and Apache Struts.

Scala is my first pick for the programming language that is most excitingly poised for the multi-core programming revolution in the age of multicore processors, cloud-computing, web 2.0 and access to all your favorite Java APIs/apps.

BOOK: Landing the Tech Job You Love

2009-07-11T00:56:00.001-05:00

I blogged earlier about The Passionate Programmer and The Nomadic Developer.

A new book just came out that seems like the perfect complement to these two: Landing the Tech Job You Love by Andy Lester (also from the Pragmatic Programmers). I've only just started reading it but so far I like it a LOT! It also received a nice review by Mike Riley in DDJ online.

Rather than cite the blurb from the book this time, I rather like the description given in the press release:

It’s tougher than ever to get that great job. Companies are more demanding and your competition is smart, tech-savvy, and resourceful. You’ve got the right skills for the job—you also need the right skills for job hunting. You need to work deliberately on your new job hunting skills, and this book can help. Old fashioned cookie-cutter job hunting skills aren’t enough: Land the Tech Job You Love gives you the background and the hard-won wisdom to leapfrog those who play by the old rules.

Andy tells us, “Life is too short for a job you don’t love. You’re not stuck—other opportunities are available for you, if you know where to look and can work the hiring process to your advantage. This book will help you get that job you love.”

In this book, you’ll learn how to find the job you want that fits you and your employer. You’ll uncover the hidden jobs that never make it into the classifieds or Monster. You’ll start making and maintaining the connections that will drive your future career moves.

Andy started writing this book years before the recession (a.k.a. “econopocalypse”) hit. He looked at the conventional wisdom and the advice available in generic books on job hunting, and found the conventional wisdom just didn’t work for programmers, system administrators, testers, and other related development positions.

He looked at everything from whether you should look for work on online job boards to whether you should lead off your resume with your objectives. Although he has definite answers for these two, he found that the answer to most questions is “it depends.” His book leads you to taking an honest assessment of what you offer and what you want in a job so that you end up in a job that is a good fit for you and your employer.

This is an important book for you to read whether you currently have a job or not. The same tactics you take to make yourself more employable will also make it easier to get promoted in your current company.

Also see Andy Lester's presentation on Effective Job Interviewing from Both Side of the Desk and his blog @ TheWorkingGeek.com for his thoughts on "Job hunting and working life for programmers, sysadmins and all other techies"

BOOK: The Economics of Iterative Software Development

2009-07-08T00:35:00.000-05:00

In the July issue of the Agile Journal I reviewed Walker Royce, Kurt Bittner and Mike Perrow's book The Economics of Iterative Software Development: Steering Toward Better Business Results. Here is an excerpt ...

The Economics of Iterative Software Development: Steering Toward Better Business Results is an important text for anyone trying to persuade management to "go iterative" as well as to anyone needing to measure & track the kinds of business results that management needs to see for a software development project.

I'll be perfectly honest: I was expecting this book to be an extremely dry and boring read, albeit full of lots of useful information densely packed in mathematical models and formulas, perhaps reminiscent of past college days reading a huge tome on socio-political economic theories. It wasn't as bad as I'd feared. Yes - it is a bit dry in places, but those are the exception rather than the rule. And there are lots of "war stories" sprinkled throughout that hold your interest and stop your eyes from glazing over.

...

All in all, The Economics of Iterative Software Development is a solid-book that is a relatively light/quick read with lots of good, practical advice on how to apply an economic model of thinking and measurement for managing and tracking business-results of an iterative project. If you need help doing that, or with communicating to your management about the need and benefits of an iterative approach, then you need to read this book!

See the full review for more details.

JEDI Programming - Just Enough Design Initially

2009-07-06T01:33:00.005-05:00

I left a comment on the "What is Missing?" entry at the Agile-in-a-Flash blog. The author's asked the questioin "What is missing?" from the stack of Agile flashcards they are developing. I responded ...

I think the "JEDI" approach is missing (any by that, I don't mean the mantra of "use the source Luke" ;-)

I think there is something missing regarding TDD and Design. Uncle Bob's three rules of TDD (and other writings) often mislead people to think that there is ZERO design up-front, as is if NOT doing Big-Design-Up-Front (BDUF) implies that therefore there is zero up-front design (NoDUF).

This is false (and Uncle Bob has even vehemently said so in The Scatology of Agile Architecture) but how does a newcomer reconcile it with the three rules of TDD? I can't write test-code without being able to invoke the thing-under-test. I can't invoke a thing if I haven't attempted to design the interface.

If I design an interface (even for a single method/subroutine) I have to have some inkling of which class/package/module it would go in, at least INITIALLY! There is some initial amount of design I do before writing a test that is both necessary and sufficient to define "just enough" of the interface of what I want my test-case to test.

So I think that is what is missing, a card called "JEDI", for "Just Enough Design Initially."

To my knowledge, this particular definition of the JEDI acronym in agile development was first used by Stephen Palmer and other FDD luminaries at www.step-10.com and featuredrivendevelopment.com (just do a Google-search on "JEDI" AND "Just Enough Design").

I also think there is a relationship between JEDI and Eric Evan's " Domain-Driven Design (DDD), Supple Design (part of DDD), as well as *some* of the so-called "Pre-factoring". But it can be a risky, slippery-slope, so it would be great to have some guidance to help us know when we've done "Just Enough Design In-front/Initially."

I suppose JEDI is a way of straddling the "appropriate range" of risk between anticipation and adaptation. I envision some kind of graph or diagram with axes ...

On the left-hand side of overanticipating we have "too much too soon" and big/all up-front design.

All the way on the right-hand-side we have "too little too late." Here you are faced with legacy-rewrites, system re-engineering, large-scale restructuring, etc.

The problem with both extremes is the creation of technical debt:

One Extreme does it by adding complex structures and behaviors too early in the cone-of-uncertainty and causes too much rework to rewrite that which was not yet certain and subject to much variability.

The other extreme does it by sheer entropy/decay/rot that results from inattention and/or negligence

In the middle we have a delicate balance that we need to strike, and which JEDI represents: "Just enough" and "Just-in-Time." The problem is that this is a range, not a single perfect point. What makes the problem harder is that the range is different for different projects, depending on a number of factors (including scale and distribution). There is a certain amount of "pay-up-front" and "pay-later" that need to be balanced with "pay-as-you-go."

I imagine this "range" represents how to find the "sweet spot" that demarcates Just Enough Design Initially:

In the middle of the scale is pure refactoring. It is strictly emergent, pay as you go just-in-time by focusing ruthlessly on keeping code clean and simple.

Minor-to-moderate restructurings (too large to be refactorings, too small to be total re-writes) are to the right. Sometimes larger systems and/or systems constructed by multiple-teams can do a great job at refactoring and still not be able to avoid the need for minor-to-moderate restructuring, particular for aspects of the design that cut across teams and architecture and functionality.

Just to the left of refactoring would be so-called "pre-factoring", where you have enough experience refactoring that you are able to apply basic application of encapsulation, DRY, separation of concerns, etc. without adding premature abstraction of inessential complexity. This is hard to get right, and has risk associated with it. But it does get progressively less with the better judgment that comes experience and practice.

To the left of pre-factoring would be the subset of DDD known as Supple Design. And the rest of DDD would be to the left of supple-design.

Somewhere in this continuum might be "segments" corresponding to specific practices or techniques besides DDD and refactoring, such as clean code, simple/incremental design, and evolutionary architecture.

The trick of knowing the "just enough" range lies not just in experience and discipline, but also in understanding your context and your "lead-time". Lead-time in particular dictates how soon in advance you need to be able to think and anticipate (and at what level of detail). The shorter the lead/cycle-time, the less you need to anticipate and prognosticate.

So I would see JEDI as a card that somehow is able to depict this continuum between anticipation and adaptation and where these various techniques fall on it, and the factors/tradeoffs that help you identify the right "range" for yourself. (And of course there is a close relation to technical debt and the cost-of-change curve :-)

Anyone ever seen a diagram that bears any resemblance to what I'm thinking of here?

Emergent Design and Evolutionary Architecture - Resources

2009-07-02T02:02:00.018-05:00

As a bit of a follow-up to my earlier posting on Technical Debt - Definition and Resources I gathered some resources on the subject of Evolutionary Architecture and Emergent Design (which is closely related to refactoring, restructuring and reengineering).

Emergent Processes and Emergent Models of Architectural Practice, by Tom Wiscombe of emergentarchitecture.com
Neal Ford's series of IBM developerWorks articles on Evolutionary Architecture and Emergent Design
Chris Sterling's (upcoming) book on Architecture in an Agile Organization
Articles on Emergent Architecture, from InfoQ.com
Competing in the Era of Emergent Architecture: The Case of Packaged Software Industry, Proceedings of the 40th Annual Hawaii International Conference on System Sciences (2007)
Any of the fine presentations by Ryan Shriver, over at theagileengineer.com
Dean Leffingwell's writings on Agile Architecture and Intentional Architecture
Jon Kern's Just Enough Early Architecture to Guide Development, from TechnicalDebt.WetPaint.com
OOPSLA '09 workshop on "Architecture in an Agile World"
Software Architecture and Agile Software Development - An Oxymoron? by Philippe Kruchten
Agile Architect by Philippe Kruchten
Is Design Dead? by Martin Fowler (see the section "Growing an Architecture")
97 Things Every Software Architect Should Know (the book)
Extending an architecture as it earns business value by Alistair Cockburn
Agile Architecture - How much is enough?, by Eoin Woods
The Agile Architect site (including the role of the agile architect)
Articles at blog.softwarearchitecture.com
Agile Architecting by Erik Philippus
Eric Evans' work on Domain-Driven Design, see in particular the DDD Pattern Summaries and the InfoQ eBook Domain Driven Design Quickly (also this Step-by-Step Guide to DDD)
Scott Ambler's Scaling Agile Development via Architecture from the Agile Journal
Lean Software Architecture, from Jim Coplien and Gertrud Bjornvig
Agile Architecture and Agile Architecture Requires Modularity, by Kirk Knoernschild
Product Line Architectures and Agile Development, M.Ali Babar et.al.
Going Agile? Beware of Architecture-Related Changes and Challenges, M.Ali Babar et.al.
The Evolutionary Architecture Practice from the Eclipse Process Framework
DevJam's David Hussman on Architecture and Agility
Architecture Meets Agility, by Hakan Erdogmus
Toward and Evolutionary System of Systems Architecture, by Scott Selberg & Mark Austin, INCOSE 2008
Guidelines for Lean Model-Based (System) Architecting and Software Engineering
Systems Engineering and Architecting Challenges: Application to Agile Development, by Murray Cantor
The (freely available online) book Object-Oriented Reengineering Patterns, by Serge Demeyer, Stéphane Ducasse and Oscar Nierstrasz
The System Reengineering Patterns Project, by Perdita Stevens et.al. from Heriot-Watt University in Edinburgh (also see the SEI page on software reengineering)

Embracing Change - quotable quotes on change and uncertainty

2009-06-28T00:52:00.012-05:00

Every now and then I come across a good quote about embracing the fact that change and uncertainty are an essential inherent reality of software development. Here are the ones I like so far. Do you have another one you like? If so, leave a comment and share it with me.

It’s inevitable that requirements will change. Business needs evolve, new users or markets are identified, business rules and government regulations are revised, and operating environments change over time. In addition, the business need becomes clearer as the key stakeholders become better educated about what their true needs are.
-- Karl Wiegers, Cosmic Truths about Software Requirements

The growing unpredictability of the future is one of the most challenging aspects of the new economy:

Turbulence—in both business and technology—causes change, which can be viewed either as a threat to be guarded against or as an opportunity to be embraced.
Rather than resist change, the agile approach strives to accommodate it as easily and efficiently as possible, while maintaining an awareness of its consequences.

Facilitating change is more effective than attempting to prevent it.

Learn to trust in your ability to respond to unpredictable events; it's more important than trusting in your ability to plan for disaster.
Agile processes harness change for the customer's competitive advantage.

-- Martin Fowler and James Highsmith, On the Agile Manifesto

Orders must change rapidly in response to change in circumstances. If one sticks to the idea that once set, a plan should not be changed, a business cannot exist for long.
-- Taiichi Ohno, Toyota Production System

It’s not the strongest who survive, nor the most intelligent, but the ones most adaptable to change.
-- attributed to Charles Darwin

Doubt is not a pleasant condition, but certainty is absurd.
-- Voltaire

It is not necessary to change; survival is not mandatory.
-- W. Edwards Deming

When the rate of change outside exceeds the rate of change inside, the end is in sight.
-- Jack Welch

The Futility of Fighting Change: Requirements, technologies, teams, priorities, companies, usage, users, everything will change. There are things you cannot know until you know them. Give up. Change is inevitable. Our process must be a process, therefore, of change.
-- Scott L. Bain The Nature of Software Development, May 2008

Uncertainty is inherent and inevitable in software development processes and products.
-- H. Ziv and D.J. Richardson, The Uncertainty Principle in Software Engineering, Aug 1996

It will be important to organize future projects with enough agility to be able to adapt to the additional emergence of new sources of unpredictable change.
-- Barry Boehm, Making a Difference in the Software Industry, March 2008

When we program, we transform a poorly understood problem into a precise set of instructions that can be executed by a computer.
When we think we understand a program?s requirements, we are invariably wrong.
When we do not completely understand a problem, we must research it until we know that we understand it.
Only when we truly understand a problem can we develop a superior product to address that problem.
What the users think they want will change as soon as they see what we develop.

-- Watts Humphrey, Some Programming Principles--Requirements, January 2003

There is a very fancy technical term biologists use to describe completely stable systems. This highly sophisticated technical term is the word “DEAD!” Change is not the enemy – stagnation is!
-- from The Unchangeable Rules of Software Change

Requirements changes late in the lifecycle are competitive advantage, IF you can act on them!
-- Mary Poppendieck

Becoming agile means accepting uncertainty about the future as a way of dealing with the future. Any project development effort should be a balance between anticipation (planning based on what we know now) and adaptation (responding to what we learn over time).
-- James Highsmith, Embrace Uncertainty

Scrum’s Uncertainty Principle is: Customers don’t know what they want until they see it, and they always reserve the right to change their mind.
-- Jeff Sutherland, Emergence of Essential Systems

Systems requirements cannot ever be stated fully in advance, not even in principle, because the user doesn’t know them in advance--not even in principle.

To assert otherwise is to ignore the fact that the development process itself changes the user’s perceptions of what is possible, increases his or her insights into the applications environment, and indeed often changes that environment itself.

We suggest an analogy with the Heisenberg Uncertainty Principle: any system development activity inevitably changes the environment out of which the need for the system arose. System development methodology must take into account that the user, and his or her needs and environment, change during the process.
-- Dan McCracken and Michael A. Jackson, ACM SIGSoft, 1982

Is your favorite quote about software change/uncertainty missing from the above? Leave a comment and tell me about it!

Technical Debt - Definition and Resources

2009-06-24T00:47:00.042-05:00

I ran across a few really good papers on the subject of technical debt that are fairly comprehensive in their treatment of not just what it is, but also how to manage it:

Technical Debt and Design Death: How to ensure you can deliver business value in the future as well as in the present; by Kane Mar and Michael James
Managing Technical Debt - a whitepaper, by Steve McConnell (requires registration, which is free, but also see the blog-entry 10X Software Development - Technical Debt)
Design debt economics: A vocabulary for describing the causes, costs, and cures for software maintainability problems; by John Elm, IBM developerWorks, June 2009
Managing Technical Debt; by Tom Brazier
Software Debt - Depreciation of Value for Software Assets and Technical Debt - Reducing Internal Quality for Short-term Gain, by Chris Sterling, from his forthcoming book Managing Software Debt (also see his video presentation)

For those not already well-versed on the subject, Technical Debt [a.k.a. Design Debt] occurs when our software becomes difficult or risky to change, and takes increasingly more time & effort to evolve. Technical debt represents the cost of the accumulated amount of rework that will be necessary to correct and/or recover from the deviation between:

the current design of the system, versus ...
a design that is minimally complex yet sufficiently complete to ensure correctness & consistency for timely delivery.

This effort grows more than linearly over time as a system becomes bigger and more complex.

The economic impact of technical debt is directly related to the cost of complexity and its resulting “friction” against the velocity of the development team (and, subsequently, upon the ease of system evolution).

Technical debt can be caused by under-engineering just as much as it can be caused by overengineering (overdesigning). It is a difficult, delicate and dynamic balancing act to achieve the necessary and sufficient amount of design to implement only the essential complexity required by system:

Sometimes we knowingly (under great pressure) do something the "quick & dirty" way, with the intent to "do it right" later (but not too late).
Sometimes we try to do too much to soon, and elaborate or implement details of the requirements when we and our customers/users haven't yet learned enough about the true needs of the system.
Sometimes we haven't yet learned enough about good design, and unintentionally violate design principles, resulting in undesirable dependencies that make code or other work-products hard to change.
Sometimes we neglect to properly "tend" to the design and don't give it the necessary amount of ongoing care and feeding needed to keep it "fit" and "supple."

But whether it is a deviation between principles and practice (knowingly or unknowingly), guessing incorrectly, anticipating too early, neglect, poor quality, or even just the laws of software evolution, we must make plans and take action to deal with the inevitable entropy of evolution or else we will sink too far into technical debt.

That's just my two cents on the subject of course. What do others have to say about it?

Wikipedia defines Technical Debt by referring to the words of Ward Cunningham, who first drew the comparison between technical complexity and debt in a 1992 experience report:

"Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."

On the C2 Wiki, Ward defined Technical Debt as:

“Technical Debt includes those internal things that you choose not to do now, but which will impede future development if left undone. This includes deferred refactoring. Technical Debt doesn't include deferred functionality, except possibly in edge cases where delivered functionality is "good enough" for the customer, but doesn't satisfy some standard (e.g., a UI element that isn't fully compliant with some UI standard).”

Martin Fowler's definition of Technical Debt is also frequently cited:

"Doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future."

In his introduction to Refactoring to Patterns, Joshua Kerievsky quite simply defines design debt as follows:

Design debt occurs when you don't consistently do three things.
Remove duplication.
Simplify your code.
Clarify you code's intent.
Few systems remain completely free of design debt. Wired as we are, humans just don't write perfect code the first time around. We naturally accumulate design debt ....

Due to ignorance or a commitment to "not fix what ain't broken," many programmers and teams spend little time paying down design debt.... In financial terms, if you don't pay off a debt, you incur late fees. If you don't pay your late fees, you incur higher late fees. The more you don't pay, the worse your fees and payments become. Compound interest kicks in, and as time goes on, getting out of debt becomes an impossible dream. So it is with design debt.

James Shore describes Design Debt in much the same manner:

When the cost of change increases, it's because the design quality is decreasing. Since retiring or rewriting software means writing off a huge investment, you wouldn't expect any team to let design quality deteriorate to that point. Yet it happens all the time. Why?

Design debt" explains the problem. When a team is working under pressure, they take shortcuts that compromise design quality. It's like taking out a high-interest loan. The team gets a short-term boost in speed, but from that point forward, changes are more expensive: they're paying interest on the loan. The only way to stop paying interest is to pay back the loan's principle and fix the design shortcuts.

Steve McConnell defines Technical Debt as follows:

“Technical Debt” refers to delayed technical work that is incurred when technical short cuts are taken, usually in pursuit of calendar-driven software schedules. Just like financial debt, some technical debts can serve valuable business purposes. Other technical debts are simply counterproductive. The ability to take on debt safely, track their debt, manage their debt, and pay down their debt varies among different organizations. Explicit decision making before taking on debt and more explicit tracking of debt are advised.

Mary Poppendieck gives a definition of Technical Debt in her upcoming book Leading Lean Software Development:

“All successful software gets changed. So if we think we’re working on code that will be successful … we need to keep it easy to change. Anything that makes code difficult to change is technical debt. Just like any other debt, the cost of paying off technical debt gets more and more expensive over time. … Technical debt drives the total cost of software ownership relentlessly higher … eventually we will have to pay it off or the system will go bankrupt.”

Kane Mar, in Technical Debt and Design Death, describes technical debt and later likens it to the effects of entropy:

Technical debt is simply defined as deferred work that is not directly related to new functionality, but necessary for the overall quality of the system. Examples of this include delaying upgrades to necessary tools and frameworks, delaying the refactoring of overly complex system components, and so on. If this technical work is necessary for the health of the system, then what happens when it is ignored for too long? What happens when several layers of cruft accumulate and there are no unit tests to aid refactoring? ...

In thermodynamics, “entropy” refers to the randomness of the components of a system. When the term is applied to software it is considered a measure of disorder. Thus entropy in software is the result of changes made to the code base, including bug fixes, updates to existing functionality, and the addition of new functionality. But over a period of time, these small changes can snowball to create a system that is difficult to change, overly connected to external systems, and lacks clear delineation of functionality.

Chris Sterling, in his upcoming book on Architecture in an Agile Organization, distinguishes between Software Debt and Technical Debt:

Software debt accumulates when focus remains on immediate completion while neglecting changeability of the system. The accumulation of debt does not impact software delivery immediately. At first it creates a sense of increased feature delivery with management, business stakeholders and the team. Business stakeholders respond well to the pace of delivered functionality. What they don’t understand is that this only represents an illusion of earlier returns on their investment.
...
This allows both the business and software delivery teams to live in the illusion of status quo far longer than they should. At some point previously small forms of decay in the system become large enough to affect our software delivery to the point that working harder and longer doesn’t result in success.
...
Debt is made glaringly visible when the team works on stabilizing the software functionality late in the release cycle. Integration, testing, and bug fixing is unpredictable and does not get resolved adequately before the release. People involved in the project stay late working to get the release out the door. It is now too late to pay back the debt accrued during the feature development.

The following sources constitute what I call software debt:
Technical Debt: those activities that a team or team members chooses not to do now and will impede future development if left undone
Quality Debt: diminishing ability to verify functional and technical quality of entire system
Configuration Management Debt: integration and release management become more risky, complex, and error-prone
Design Debt: cost of adding average sized features is increasing to more than writing from scratch
Platform Experience Debt: availability and cost of people to work on system features are becoming limited

Over at the Agile-in-a-Flash blog, Jeff Langr and Tim Ottinger provide a flashcard of tips & truths about technical debt, along with a more detailed discussion about its nature and origins. They distill several nuggets of wisdom like:

Incurring technical debt means your velocity slows down and you will deliver less value
The cost of getting out-of-debt is compounded over time: the longer you wait, the faster it grow!
If you plan to incur technical debt, the persons responsible must have a workable plan to pay it off!
“Interest only” payments won’t improve things
Pay early, pay often, and pay-as-you-go. (The only other options are bankruptcy or death.)
Remember: Those with the worst debt problems often have the most difficulty imagining a life without borrowing!

So how might we properly measure/monitor and account for technical debt? Well, some of the above papers I mentioned have some good ideas. Some that address it more explicitly are:

Israel Gat's Agile Executive Articles on Technical Debt

Monetizing Technical Debt (also see other InfoQ.com articles on Technical Debt)

An Approximate Measure of Technical Debt by James Shore
Evaluate your technical debt with Sonar (SONAR is a free open-source tool)
Martin Fowler writes about Estimated Interest
In Does Cost Accounting Cause Crappy Code? Amr Elssamadisy suggests using throughput accounting (from TOC) instead of cost-accounting for understanding the cost of design debt

Here are a number of other very good resources on technical debt culled from blogs, articles, and presentations over the years. Some of them describe it, while others delve into methods for managing it and paying it off:

Repaying Technical Debt, by Simon Baker
Software Entropy: Don’t Tolerate Broken Windows, and Zero-Tolerance Construction by Andy Hunt and Dave Thomas
The Cost of Complexity and Coupling, by Mark Heath
Coding Horrors: Paying Down your Technical Debt, by Jeff Atwood
What Testers Can Do about Technical Debt (Part1 and Part 2), by Johanna Rothman (also see Johanna's earlier essay Climbing Out of Technical Debt)
Technical Debt - How not to ignore it!, by Henrik Kniberg
Get out of Technical Debt Now!, by Andy Lester
Continous Refactoring and the Cost of Decay, by Patrick W. Welsh
Agile Advice on Technical Debt, by Mishkin Bertieg
A Fresh Look at Technical Debt, an InfoQ.com summary of a Technical Debt Workshop
Managing your Analysis debt, by Ellen Gottesdiener

Dissecting Technical Debt, an InfoQ.com summary from Agile techniques

Paying Down Technical Debt, by Brandon Savage
Refinance your Technical Debt Just Like Your Mortgage
Managing Software Debt, by Chris Sterling (also see video presentation)
Fighting technical debt with the Wall of Pain, by Jimmy Boggard
Development Psychology, technical debt and the next feature syndrome
Where does technical debt come from?, by David Christiansen
Technical Debt - deficit spending for geeks, by Dave Churchville
Five Kinds of Technical Debt, by Ian Cartwright
Technical Debt: The threshold of acceptable pain, by (Technoetic) Steve Bate
Technical Debt: Transparency and Customer Communication, by Ron Jeffries
Technical Debt Quadrants, by Martin Fowler (also see Design Stamina Hypothesis)
We must ship now and deal with the consequences, by (Uncle) Bob Martin - a response to Martin Fowler's article above
Don't Enron your software project, by Aaron Erickson
Technical Debt: What is it? The Warning Signs and Paying it Off, by Catherine Powell

Resources on Self-Organizing Teams for Agility

2009-06-20T01:08:00.016-05:00

In the past several blog-entries I've been focusing on the agile principle of self-organization, what it means, and what it implies for teams. So far, I've written about Agile Self-Organization versus Lean Leadership, Self-Organization and Complexity, and Agile Self-Organizing Teams.

For some online resources (articles and presentations) on self-organization and self-organizing teams, I recommend the following:

InfoQ.Com articles: High-Performance Teams – Avoiding Teamicide, Coaching Self Organizing Teams, and Ensuring Success for Self Organizing Teams
Leading a Self-Organizing Team, PDF presentation by Mike Cohn given at the Jan 2009 Dallas Agile Project Leadership Network (APLN)
Flock-theory applied, by James Brett -- a look at Flock Theory by D.Rosen for answers to creating highly optimized, self-organised Scrum teams
Exploring Self-Organizing Software Development Teams, by Diana Larsen
Agile Processes and Self-organization, by Ken Schwaber
Understanding Self-organizing Teams in Agile Software Development, from 2008 Australian Software Engineering Conference (ASWEC2008)
The Manager's role in Agile, and 7 Agile Coach Failure Modes, by Lyssa Adkins and Michael Spayd
Scum vs Scrum: Emergent behavior in slime molds and software development teams, by Matt Truxaw
Agile Software Development and Complex Adaptive Systems, a 4 part series of articles by Michael Dubakov:
- part1: Introduction to Agile Software Development and CAS
- part2: Software Development is a Complex Adaptive System - No Doubt
- part3: The Edge of Chaos and Hyper-Productive Software Development Teams
- part4: Simple Rules, Complex Systems and Software Development
Holacracy - The Self-Organizing Enterprise
Steering from the Edges, by Sanjiv Augustine et.al.
Roots of Scrum: Takeuchi and Self-Organizing Teams, by Jeff Sutherland
Learning is the Bottleneck, by Amr Elssamadisy & Deborah Hartmann
Leadership is not obsolete in Self-Organizing Teams, by Mary Poppendieck
More InfoQ.com articles on Self-Organizing Teams
Polyanna Pixton's papers/presentations on Agile leadership & collaboration
When to stand back, when to step-in (Managing self-organizing teams), by Esther Derby
What is the best leadership style for a software team? by Andriy Solovey (also see Evolutionary Software Architecture and Self-Organizing Teams: The Army versus Jim Highsmith
Self Organization in Scrum and Self-Organization & Shock Therapy, presentations by Jeff Sutherland (also see the corresponding video Self-Organization - The Secret Sauce for Improving Your Scrum Team)
Of ants and men: self-organized teams in human and insect organizations
The Meaning of Self-Organization in Computing and The Science of Self-Organization and Adaptivity, by Francis Heylighen
Mishkin Berteig on Team Self-Organization
Swarming: The Birds, Bees and Agile, by Tom Perry and Dhaval Panchal
Limits of Self-Organizing Teams, by Alan Kelly
Bureaucracy & Organizational Politics: Emergent Characteristics of Structure, an essay from systems-thinking.org
Jurgen Appelo on Self-Organizing Teams
Systemic Analysis of A Team’s Self- Organizing processes: some insights into the knowledge management
Be Empowered (That’s an Order !) “Experience the Dynamics and the Paradoxes of Self-Organizing Teams”, by Laurent Bossavit and Emannuel Gaillot
Conditions for Self-Organizing in Human Systems by Glenda Holladay Eoyang (Ph.D. Thesis)
Series of articles on Group Coherence for Agile/Project Teams:

Team Effectiveness in Complex Organizations
Ensuring Success: A Model for Self-Managed Teams, by Lori Silverman & Annabeth Propst
Using Emergence to Take Social Innovation to Scale, and Leadership of Self-Organized Networks: Lessons from the War on Terror, by Margaret Wheatley et.al.
Complexity, Organizations and Change, by Elizabeth McMillan
Managing the Unknowable: Strategic Boundaries Between Order and Chaos in Organizations
The Universal principles of Self-organization and the Unity of Nature and Knowledge, by Alexei Kurakin
Classification of Emergence and its Relation to Self-Organization (also see related papers on emergence by Fabio Boschetti)
Wikipedia entries on Self-Organization and Emergence (also Wapedia entries on Self-Organization and Emergence)
Self-Organizing Systems FAQ
10 Questions about Emergence
Self-Organization Dynamics
Patterns in Nature
Order Out of Chaos, Ilya Prigogine
Using Positioning Theory to Make Change Happen, by Lionel Boxer

I'll be taking a break on this topic (Self-Organization) for awhile, and then returning to it again later to talk about Swarm Behavior, Collective Intelligence, Social Creativity, Group Coherence, Group Decision-Making and Holocracy/Sociocracy.

Agile Self-Organizing Teams

2009-06-16T00:16:00.011-05:00

The previous blog-entry on self-organization was lots of jargon and technical mumbo jumbo that didn't say too much about what that means for teams of people. So let's shift from talking about self-organizing systems in complexity science to talking about how it applies to self-organizing teams in an agile context.

A self-organizing team is a team that is led and organized by it's members, to attain goals and objectives specified by management within the constraints of its environment:

Management can shape and "nudge" the team and its members, but management doesn't try to dictate the details of "what" the solution is nor the process of how to create it.
The team is responsible for not only leading and organizing itself to achieve its goals, but also to monitor and adapt its behavior to correct/improve its own performance.
This means the team can change how it leads and organizes itself in order to respond to feedback and constraints from its environment, which also implies that ...
There is no single central "leader" for the team over the lifetime of the team/project - the "leader" is not a static assignment, but rather a dynamic role
So the person(s) leading any given moment may change, depending on the particular decision, activity, or problem being addressed in any particular context/situation.

By themselves, self-organizing teams are neither "good" nor "bad." They simply "are." They require a supporting management environment (the "fitness landscape") and organizational culture that establishes, communicates, rewards and reinforces the "right" set of values and principles. Without supportive management and the proper leadership culture, there is a very high likelihood that a self-organizing team may be unable to create good results or effective processes (or both). In fact, it's not uncommon for a newly formed & "empowered" self-organizing team to fall into many of the same dysfunctional patterns of behavior that it was most trying to escape from within the "management" that only recently "empowered" the team.

An "agile team" is (supposed to be) a self-organizing team that is guided by the agile values and agile principles (given by the agile manifesto) and is supported by a trusting and empowering style of management. With management supporting their agile values/principles, Agile teams "self-organize" to collectively decide and do what is needed in order to: make and meet commitments, develop a quality product, respond to feedback, and adapt to changes.

So an Agile Self-Organizing Team is:

Autonomous: There is no single central decision-making authority. Control is distributed collectively to the team.

Adaptive: The team dynamically adjusts as needed across roles, functional specialties, and other boundaries, in order to solve their own problems and improve their own performance.

Accountable: The team collectively shares responsibility for results, and members hold each other accountable for outcomes.

Here are some choice quotes regarding self-organizing teams ...

“The team makes most decisions, while every member could step in and become leader in specific areas and situations. People are highly capable, committed and self-driven.”
—Andriy Solovey, What is the best leadership style for the software team?

“This causes a shift in the roles of managers from planning, controlling, directing, and managing to new roles like building trust, facilitating and supporting team decisions, expanding team capabilities, anticipating and influencing change.”
—Diana Larsen, Exploring Self-Organizing Software Development Teams

"Responsibility-Based Planning and Control: Respecting people means that teams are given general plans and reasonable goals and are trusted to self-organize to meet the goals. Respect means that instead of telling people what to do and how to do it, you develop a reflexive organization where people use their heads and figure this out for themselves."
—Mary Poppendieck, Implementing Lean Software Development

In Learning is the Bottleneck, Amr Elssamadisy & Deborah Hartmann write:

"Human psychology aspect adds that self-organized teams:
are more responsible for end results, self-disciplined and self-driven
avoid dependency on the formal leader qualities
motivated, initiative and willing to act
enjoy work more
better insured against groupthink, conformity and diffusion of responsibility
not shifting judgment and decisions to others, better in finding alternative and balancing options
every member is in charge, ready to step in as a leader and have incentive to develop leadership skills
A self-organized team is possible when people carry shared purpose, principles and values. They support and respect each other. And they want to succeed. The [Agile] team works together to respond to changes that happen together. They collectively do what needs to be done to build the software."

In his 2001 paper Agile Processes and Self-Organization Ken Schwaber wrote:

"Agile processes employ self-organizing teams to handle the complexity inherent in systems development projects. A team of individuals is formed. They organize themselves into a team in response to the pressure of a deadline, reminding me of the saying, "Nothing focuses the mind like a noose!" The pressure cooker of the deadline produces cooperation and creativity that otherwise is rare. This may seem inhumane, but compared with non-agile practices for dealing with complexity, self-organization is a breath of fresh air."

This is what Kevin Kelly wrote about that problem in his book Out of Control:

"When everything is connected to everything in a distributed network, everything happens at once. When everything happens at once, wide and fast moving problems simply route around any central authority. Therefore overall governance must arise from the most humble interdependent acts done locally in parallel, and not from a central command."

Roger Lewin wrote in Complexity: Life at the Edge of Chaos:

"Complexity science implies that CEOs and managers must give up control -- or rather, the illusion of control -- when they are trying to lead their organization to some goal. But they do need to create the environment in which creativity can emerge. The message of complexity science is not simply to stand back and wait for the right solutions to emerge. Too little control is just as misguided a business strategy as too much. Some structure is necessary. The degree and nature of control that CEOs establish in their companies strongly influences what emerges, in terms of culture, creativity, and adaptability."

Mishkin Berteig writes in Team Self-Organization:

"In agile teams, this concept of self-organization is taken quite far. Team members collaborate to get work done. No one orders a team or an individual to do specific work. The team members volunteer for work that they see needs doing, even if it is not something that is in their area of expertise. An agile team is constantly promoting learning in its people. Agile teams are also cross-functional so that the team can get work done without relying on external services. The team therefore represents a complete work unit capable of taking a function valuable to customers from start to finish, from idea to deployment."

In Why Agile Principles are Important, Simon Baker writes:

An experienced agile software development team is a highly social group that is self-organising around these principles and acts with coordination and collective behaviour. This collective behaviour comprises:

Collective mind where individual team members develop shared understandings of the team's tasks and of one another, and come to understand how their work contributes to the work of the team thereby facilitating team performance.

Swarm intelligence which gives a team the ability to adapt to changes, and robustness which enables them to still perform and deliver even when one or more members fail.

From Of ants and men: self-organized teams in human and insect organizations

To cope with today’s complex, fast-paced, and ever-changing business environment, companies need to shift their overall structure to produce adaptive, highly responsive organizations. The use of teams, particularly self-organized teams with their reactive, emergent properties, may be one way of achieving this goal.

In other words, insect societies often harness the power of self-organization such that with the appropriate set of feedbacks, interindividual interactions, and proximate mechanisms, group-level adaptive behavior simply emerges. No one directs the foragers where to find food, the network of trails and interactions takes care of that; individuals are not allocated to tasks, the reverse is true: the tasks allocate the workers.

McMillan-Parsons (1999) found that the teams fitted Stacey’s (1996) description of self-organizing groups or teams as ones that arise spontaneously around specific issues, communicate and cooperate about these issues, reach a consensus, and make a committed response to these issues. Further, ‘research suggested that self organizing teams have a strong sense of shared purpose, strong personal commitment, display creative and spontaneous behaviors, have high levels of energy and enthusiasm, and that an inherent order emerges from their activities’.

Importantly, in self-organizing teams the members self select and there is no-one checking to see if they have the necessary range of attributes. In her study, McMillan (1999) discovered that members of the self-organizing teams studied learned new skills and developed new attributes to meet the needs of the team.

One characteristic of such organizations is adhocracy. These large, mature yet high-performing companies manage to generate the flexible and adaptive properties of smaller entrepreneurial organizations—in short, to “be big and yet to act small at the same time”. Using teams is one key means of achieving that, for, as Flory (2002, p. 9) remarks, self-managed teams, ‘are fast moving, fast learning groups, flexible, highly autonomous and have a well-developed pro-active attitude and sense of responsibility. These characteristics are the very reason they are brought into life as answers for organizations to respond to a fast moving world.’

Self-managed Teams	Self-organized Teams
Part of formal organization structure	Not part of formal organization structure
Formal, temporary, or permanent	Informal and temporary
Not spontaneously formed	Formed spontaneously around issue(s)
Indirectly controlled by senior management	Boundaries influenced by senior management
Managers decide ‘who’ and ‘what’	Team members decide ‘who’ and ‘what’
Replace the hierarchy	Often in conflict with or constrained by the hierarchy
Empowered by senior management	Empowered by the team’s members
Strongly shared culture	Cultural differences provoke and constrain
Some sense of shared purpose	Strong sense of shared purpose
Order created via recognized processes	Inherent order emerges
Behaviors influenced by procedures and roles	Spontaneous, creative behaviors
Strong sense of team commitment	Strong sense of personal commitment
Some energy and enthusiasm	High levels of energy and enthusiasm
Decision making is mainly a planned process	Decision making is mainly a spontaneous process
At least one member’s primary role is organizational	All members’ primary role relate to the task

In my next blog-entry I'll give links to several other resources on Self-Organizing Teams.

Self-Organization and Complexity

2009-06-14T00:17:00.013-05:00

In my previous blog-entry I talked a little about how self-organization is a key aspect of software agility. In this blog-entry I'd like to explore in more detail just what "self-organization" really means.

Self-organization comes from complexity science and the theory of complex adaptive systems (CAS). A system is "self-organizing" if, left to its own devices, it tends to become more organized over time. This is in stark contrast to the concept of entropy from the laws of thermodynamics whereby a closed system tends to move to a state of increasing disorder in the absence of outside influences.

In a complex adaptive system where self-organization occurs, we necessarily have an open system rather than a closed one. The theory goes that if a complex system possesses the necessary emergent properties in an appropriate fitness landscape, then the system will yield emergent behavior, exhibiting system-wide "patterns", that increases the "order" or organization in the system. Hence the system has become self-organizing as a result of this emergent behavior & structure.

Some of the emergent properties of self-organizing systems can include:

Autonomy,
Adaptability,
Decentralization,
Dynamic reconfiguration,
Positive & Negative Feedback
Cooperation & Information exchange

The theory of complex systems suggests that self-organized systems are:

better in selecting optimal state and local decisions
robust
effectively adapt to environment and use feedback loops
often come up with emergent and unexpected solutions
self-regulated and better cope with perturbations

According to James Highsmith,

“There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback”

The Emergent Phenomena Research Group at Brywn Mawr says:

"In self-organizing systems, global order (i.e., complex aggregate structure/behavior) results (emerges) from the local behaviors of simple agents following simple rules specifying conditions under which those agents act (interact) in such a way that the results of the agents' actions are fed back as input to the operation of the rule system at some future time step. "

In The Science of Self-organization and Adaptivity, Francis Heylighen writes:

Self-organization is basically the spontaneous creation of a globally coherent pattern out of the local interactions between initially independent components. This collective order is organized in function of its own maintenance, and thus tends to resist perturbations. This robustness is achieved by distributed, redundant control so that damage can be restored by the remaining, undamaged sections. ...

Self-organizing systems are characterized by: distributed control (absence of centralized control), continual adaptation to a changing environment, emergent structure from local interaction, robustness/resiliency (able under change and can quickly repair, correct, adapt/adjust), non-linearity, feedback (both positive and negative), self-sufficiency and closure/coherence. ...

Organizational closure turns a collection of interacting elements into an individual, coherent whole. This whole has properties that arise out of its organization, and that cannot be reduced to the properties of its elements. Such properties are called emergent.

Every self-organizing system adapts to its environment; Systems may be called adaptive if they can adjust to such changes while keeping their organization as much as possible intact.

From Software Development is Complex Adaptive System. No Doubt

Emergence is the way complex systems and patterns arise out of a multiplicity of relatively simple interactions. Small actions of agents lead to unexpected emergent system behavior and it is impossible to predict system behavior based on the behavior of an individual agent. Small errors pile up and may cause huge problem.

There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback.

A system in equilibrium does not have the internal dynamics that enables it to respond to the environment and will slowly (or quickly) die. A system in chaos stops functioning as a system. The most productive state to be in is at the edge of chaos where there is maximum variety and creativity, leading to new possibilities. A set of simple rules allows edge of chaos and powers creativity, flexibility and success.

And last but not least, from Conditions for Self-Organizing in Human Systems :

Self-organization is the spontaneous generation of order in a complex adaptive system. It is the process by which the internal dynamics of a system generate system-wide patterns. Some of the emergent patterns in a self-organizing system are coherent, and others are not. Coherence is a state of the system in which:
Meaning is shared among agents.
Internal tension is reduced.
Actions of agents and sub-systems are aligned with system-wide intentionality.
Patterns are repeated across scales and in different parts of the system.
A minimum amount of energy of the system is dissipated through internal interactions.
Parts of the system function in complementary ways.
System-wide patterns in which the parts are aligned and mutually reinforcing (coherent) are more stable than other self-organized patterns. Because of the mutually reinforcing dynamics of a coherent pattern, the effort required to change the pattern is greater than the effort to maintain it, so coherent patterns are more stable than incoherent ones. When the system reaches a state of coherence ... tensions within the system are reduced, and the available energy of the system is aligned and focused on system-wide behaviors, rather than diverse and disruptive behavior of individual agents or sub-system clusters. ...

In human systems, the process of self-organizing is particularly important. Teams, institutions, and communities include individuals or groups of individuals that function as agents in self-organizing. As the agents interact, patterns of behavior emerge over time.These patterns form and reform spontaneously and continually at multiple levels within the system. Individuals work together to form teams. Ethnic identity groups establish relationships and micro-cultures. Functional departments engage with each other to do the work of the organization. At all of these levels, agents interact naturally to form patterns of system-wide behavior. ...

The CDE model is a set of the three conditions for self-organizing of human systems: Container, significant Difference, and transforming Exchange. The path, rate, and outcomes of self-organizing processes are influenced by these three conditions, which are co-dependent such that the function of each of the conditions depends on the others in nonlinear interactions in the system. A change in any one of the conditions results in a change in the other two over time.

Also see the following:

Wikipedia entries on Self-Organization and Emergence
Self-Organizing Systems FAQ
10 Questions about Emergence
The Meaning of Self-Organization in Computing and The Science of Self-Organization and Adaptivity, Francis Heylighen & Carlos Gershenson
Self-Organization Dynamics
Patterns in Nature
Order Out of Chaos, Ilya Prigogine
Evolutionary Software Architecture, or Why Developers are not Janitors, Andriy Solovey

In my next blog-entry I'll talk specifically about self-organizing teams. Some specific characteristics and/or results of self-organization that I'll be delving into more deeply in subsequent blog-entries are:

Swarm Behavior (a.k.a. Swarm Intelligence)
Collective Intelligence (a.k.a. Collective Mind)
Social Creativity
Group Coherence

Agile Self-Organization versus Lean Leadership

2009-06-12T00:52:00.005-05:00

Getting back to the agility cycle ... recall that I started with the business agility cycle and used that to derive the software agility cycle. There isn't a great deal of difference between the first two steps of the business-agility cycle and the software-agility cycle, other than the fact that much of the former takes place at a higher-level of organizational management and strategy.

The biggest difference between the two cycles happens after those first two steps. In the business agility cycle we then decide-communicate-act, suggesting that it is the higher-ups who make and communicate the decisions and the lower-end of the organizational food chain (the "worker bees") that execute the organization's strategic solution.

If that seems a bit command-and-control, its because it is (a bit). It's also likely necessary in larger organizations where it is virtually impossible to avoid having 2 or more layers of management. There, the "communicate & act" steps are really the process of providing focus, and communicating objectives and constraints to the knowledge workers, and letting them apply principles of collaboration and self-organization to solve the problem (like in the software-agility cycle).

So the key difference between business-agility and software-agility is the extra emphasis of the latter on "the people factor" and on the notion of dynamic self-organization of knowledge-workers as empowered, self-organizing teams. This difference between agility at the business-level versus software-level is also the key difference between Lean-vs-Agile:

Lean focuses more on flow whereas Agility focuses more on adapting to change (this is true of both software agility and business-agility). As it turns out, focusing on flow requires being able to adapt to change; and focusing on adaptiveness and responsiveness requires a focus on flow. So here, the end-results may be quite similar

Agile software development emphasizes a very hands-off management style of not just trusting and empowering the team(s), but often to the extreme of basically saying "leave us alone and don't interfere" to management (and in return promising extreme transparency and frequent tangible end-results).

Much of this is due to the fact that the scope of Agile development is limited to software projects and teams, whereas Lean is targeted at enterprise-wide scale across the whole value-stream.

That is not to say that Lean (or business-agility) don't emphasize trusting and empowering workers and teams (they do). But they don't have the underlying inherent attitude of "just trust us and stay out of the way." The "trust" often doesn't seem to extend as far in the other direction with Agile development (e.g. not trusting management direction to the same extent that management is asked to trust the team).

I think this stems from the fact that software agility started more at the grass-roots level, with developers and teams struggling to create good, quality software in the face of what was all too often a fundamentally broken management framework for the governance, visibility, and lifecycle of software development projects. Because they were working and trying to get support from the bottom-up, they needed to be shielded from and unshackled by all the dilbert-esque "pointy haired managers" and their big, bad governance framework with its "evil" waterfall lifecycle.

And in that context, the advice of "just get out of the way and trust us and we promise to deliver working software every 2-4 weeks" is actually a very effective strategy to employ. When management doesn't "get it", their attempts to steer, direct and intervene are often the equivalent of "friendly fire" (which, despite well intentions, still yields calamitous results.)

But Lean comes from a history of a much more enterprise-wide scope, often even using a top-down deployment strategy. So when it asks management to trust and empower workers and teams it expects the corresponding change in its leaders and their leadership-style, and for them to still play a strong, active and participative role with their projects and teams.

Agile teams often get a "bad rap" for having this isolationist attitude toward management. And sometimes that "rap" is warranted. But when the leadership "gets it" and understands and values the "new" paradigm of why+how agile works, and why+how the "old way" didn't, then it probably is time for the leadership to play a different, more active role while still trusting and empowering teams (and still enabling self-organization). This is the view that Lean takes toward management, and is a big part of why it is better received by management and is more broadly applicable for scaling agility "up" and "out" in larger enterprises.

Value Proposition for Agility

2009-06-10T01:17:00.012-05:00

I'm sure I'm not the first person to think it, but I just came across the description of a newly published book whose title made me think about this subject. The book is:

Reading Minds and Markets: Minimizing Risk and Maximizing Returns in a Volatile Global Marketplace, by Jack Ablin and Suzanne McGee.

The title immediately made me think that this was the right language to use when communicating with business-people and senior management to describe what agility is in terms of its benefits to the bottom line.

Investopedia describes a "Value Proposition" as: "A business or marketing statement that summarizes why a consumer should buy a product or use a service. This statement should convince a potential consumer that one particular product or service will add more value or better solve a problem than other similar offerings."

So I think that "value proposition" is the right term to describe what is it about agility that I want to describe to a business-person or senior manager that should make them want to care about what agility is and why they should adopt it.

With the above in mind, here is my "value proposition" for agility:

Agility is all about harnessing the power of collaborative people and frequent delivery to:
learn & adapt to change,
minimize risk & cycle-time, and ...
maximize returns & customer-value
in a volatile global marketplace.

There! Top that! :-)

Think you have a better value-proposition for agility that is more succinct while still touching on the minimally sufficient set of key attributes? (the people-factor, frequent value-delivery, cycle-time, responsiveness to change and risk/ROI)

If so, then I want to see it! Leave a comment and let me know!

BOOK: The Art of Lean Software Development

2009-06-09T23:18:00.002-05:00

In the June issue of the Agile Journal I reviewed Curt Hibbs, Steve Jewett and Mike Sullivan's The Art of Lean Software Development: A Practical and Incremental Approach. Here is an excerpt ...

With last month's announcement of the Lean Software and Systems Consortium at the 2009 Lean & Kanban Conference, it seems fitting that this month's book is about Lean Software Development and how Agile development practices support Lean Thinking.

The Art of Lean Software Development: A Practical and Incremental Approach is an introduction to Agile software development practices through the lense of Lean thinking. The first thing you need to know about this book is who its target audience is [...] from the publisher's website for the book: "This book has a very specific purpose -- it is aimed squarely at the complete novice. The one who has been hearing all the Lean-Agile buzz, really knows nothing about it, and wants to learn more quickly to decide if they want to dig deeper (without having to read a 500 page tome)."
...
Now that we've finally gotten that out of the way ... The Art of Lean Software Development actually succeeds at its intended purpose. It is a very "lean" introduction to the subject of applying lean thinking to software development. (You can see for yourself by looking at the online excerpts of chapter 2 and chapter 4.)
...
I like just about everything that The Art of Lean Software Development has to say. It is a bit of a dry read but a very quick one. ... For those more interested in the abridged version or the line-manager's equivalent of a technical overview, The Art of Lean Software Development is a very quick and easy read that successfully introduces the history and essentials of Lean and Agile software development from a Lean thinking perspective.

See the full review for more details.

The Dynamics of Leadership-Team Behavior

2009-06-07T09:08:00.005-05:00

Interesting article in BusinessWeek from Jim Collins on the Dynamics of Team-Leadership Behavior. It's actually an excerpt from his latest book "How the Mighty Fall: and Why Some Companies Never Give In."

Anyway ... the Dynamics of Team-Leadership Behavior is divided into leadership behaviors of teams on the way up vs. on the way down:

Teams on the Way Down	Teams on the Way Up
People shield those in power from unpleasant facts, fearful of penalties and criticism for shining light on the rough realities	People bring forth grim facts—"Come here and look, man, this is ugly"—to be discussed; leaders never criticize those who bring forth harsh realities
People assert strong opinions without providing data, evidence, or a solid argument	People bring data, evidence, logic, and solid arguments to the discussion
The team leader has a very low questions-to-statements ratio, avoiding critical input and/or allowing sloppy reasoning and unsupported opinions	The team leader employs a Socratic style, using a high questions-to-statements ratio, challenging people, and pushing for penetrating insights
Team members acquiesce to a decision but don't unify to make the decision successful—or worse, undermine it after the fact	Team members unify behind a decision once made, then work to make the decision succeed, even if they vigorously disagreed with it
Team members seek as much credit as possible for themselves, yet do not enjoy the confidence and admiration of their peers	Each team member credits other people for success, yet enjoys the confidence and admiration of his or her peers
Team members argue to look smart or to further their own interests rather than argue to find the best answers to support the overall cause	Team members argue and debate, not to improve their personal position but to find the best answers to support the overall cause
The team conducts "autopsies with blame," seeking culprits rather than wisdom	The team conducts "autopsies without blame," mining wisdom from painful experiences
Team members often fail to deliver exceptional results and blame other people or outside factors for setbacks, mistakes, and failures	Each team member delivers exceptional results, yet in the event of a setback each accepts full responsibility and learns from mistakes

BOOKS: The Passionate Programmer and the Nomadic Developer

2009-06-04T00:26:00.004-05:00

Gosh, when I write/say the titles of these two books together in one line it looks like the title of some kind of computer-geek romance novella. (maybe it will sell more books that way :-)

Anyway, I'm mentioning these two books together because they both relate to subject of managing your career if you are a professional software developer, and they are both complementary to one another.

The Passionate Programmer: Creating a Remarkable Career in Software Development, by Chad Fowler, is really the revised edition of an earlier work by him under a different book title. It's from the Pragmatic Programmers, so that by itself practically guarantees its pretty darn good. The blurb for the book is pretty darn accurate too: "This book is about creating a remarkable career in software development. In most cases, remarkable careers don’t come by chance. They require thought, intention, action, and a willingness to change course when you’ve made mistakes. Most of us have been stumbling around letting our careers take us where they may. It’s time to take control. This revised and updated second edition lays out a strategy for planning and creating a radically successful life in software development." Several excerpts are available from publisher's homepage for the book.

The Nomadic Developer: Surviving and Thriving in the World of Technology Consulting, by Aaron Erickson, is a must read for anyone considering becoming a software development consultant (or by anyone who recently became one). Chapter 6 - Surviving, is available as an online excerpt.

The "blurb" for this book is a bit long so I'll include only part of it here: "There are real advantages to being a consultant. You make contacts with a lot of different people; you get exposure to many industries; and most important, unlike a software developer in the IT department for a brick-and-mortar company, as a technology consultant, you are the profit center…so long as you are billing. Consulting can be hugely rewarding–but it’s easy to fail if you are unprepared. To succeed, you need a mentor who knows the lay of the land. Aaron Erickson is your mentor, and this is your guidebook. Erickson has done it all–from Practice Leadership to the lowest level project work. In The Nomadic Developer, he brings together his hardwon insights on becoming successful and achieving success through tough times and relentless change. You’ll find 100% practical advice and real experiences–his own and annotations from those in the trenches. In addition, renowned consultants–such as David Chappell, Bruce Eckel, Deborah Kurata, and Ted Neward–share some of their hard-earned lessons."

I don't need to tell anyone that we are in tough economic times with cuts and force reductions abounding and new jobs being scarce. Seems to me that getting both of these books together makes just plain good sense for anyone in our line of work these days.

5S Qualities of Well Designed, Well-Factored Code

2009-06-01T02:21:00.008-05:00

The other day I was trying to explain to someone the properties of code that is well-factored and found myself using aliteration with 'S' words. That made me wonder if they were equivalent to Lean's "5S", which is as follows:

Seiri (Sort) - Organize the work-area, leaving only the tools and materials necessary to perform daily activities

Seiton (Straighten, Set in Order) - the orderly arrangement of needed items so they are easy to use and accessible for “anyone” to find.

Seiso (Shine) - Keep everything clean and swept. Don’t allow litter, scrap, shavings, cuttings, etc., to land on the floor in the first place.

Seiketsu (Standardize) - Creating a consistent approach for carrying out tasks and procedures. Orderliness is the core of “standardization” and is maintained by Visual Controls.

Shitsuke (Sustain) - the discipline and commitment of all other stages. Without “sustaining”, your workplace can easily revert back to being dirty and chaotic. That is why it is so crucial for your team to be empowered to improve and maintain their workplace.

Does the above apply to the structure and content of the code itself? Have you ever come across code that is truly well-factored? I don't just mean correct and that it follows coding standards, I mean the structure of the code itself not only had such clarity of thought and order, but it also had all the qualities that we like to think of that make the code changeable/malleable with ease. Here is what I think those qualities are:

Sufficient scope (content & functionality) – The code implements only that which is necessary. It doesn't have more content than needed, or more behavior or interfaces or abstractions than needed. It is "just enough" code to get the job done, while still possessing the other properties below. XP ensures this by taking the “next most important” requirement, creating only just enough content (spec, requirements, even branches) that can be implemented for current activity in the current workspace. Then write only “just enough” code to pass the test (then you refactor).

Simple, clean code – The code isnt just "Lean" in its content and functionality, but also in its structural design. Dependencies and duplication are minimized while clarity, cohesiveness, and conciseness are maximized. In XP, once the code result is “sufficient” in content & correctness, we refactor to simplify the structure and dependencies as much as possible.

Supple design – This comes from the chapter of the same name in Eric Evans Evans' book Domain-Driven Design. Supple means pliant, malleable, limber, yielding or changing readily. Code that is well-factored is at once so simple yet sufficient as to be firm yet flexible. Yet the flexibility comes not so much from what you added as from what you left out and how you organized it. It is more than just simple and sufficient, there is an inherent model or "theory" of the program inside the programmer's head, and that structure and intent are clearly conveyed and deeply realized by the code and somehow manages to incorporate the subject domain in it as well. Evans cites patterns of supple design like: Intention-Revealing Interfaces, Side-Effect-Free Functions, Assertions, Conceptual Contours, Standalone Classes, Closure of Operations, Declarative Style of Design.

Serviceable product – Making the code easy to change isn’t enough. Use coding standards too. But beyond that, ensure that what is delivered from the code is serviceable, so that it is ALSO quick & easy to (re)build, (re)test, commit/merge, stage, release, configure and install/upgrade/deploy. It’s not just the code that needs to be quick and easy to change, it is everything that needs to be done to deliver change to the consumer in order to realize its value.

Sustainable team velocity – Okay, I'm cheating a bit here because this last one is really about the process used to attain the other 4 qualities above. It's not just the code & product that needs to be sustainable (from a business-perspective and from a support/maintenance perspective) but the process that the programmers follow. On the one hand it must necessarily be disciplined, and yet it needs to be something that does not force them to work at or above their capacity for any significant time. The process should be sustainable and renewable so that the discipline is relatively easy to sustain and in fact gets easier over time, resisting "burnout" as well as the temptation to fall back into bad habits.

So that's my "5S" for code/design structure! It's not quite as pithy as saying "lean, mean, keen, clean & green." The thing is, I'm not so sure my five S-words really do map all that accurately to the 5S of Lean (and I'm not sure I should try to make them either). What do you think?

HBR on Rebuilding Trust

2009-05-29T23:50:00.009-05:00

Some of you may recall some earlier blog-entries of mine on the topic of trust:

Trust: The social virtues and the creation of prosperity by Francis Fukuyama
More Articles on Trust
Building Trust: In Business, Poliotics, Relationship and Life by Solomon and Flores, and
The Speed of Trust: The One Thing that Changes Everything by Stephen M.R. Covey

As it turns out, the current issue of Harvard Business Review is on the theme of Rebuilding Trust (follow the hyperlink for executive summaries). The article "Trust Revisited" has a roundup of the other articles in the issue that deal with trust:

The public’s trust in business leaders has never been weaker. According to the Edelman Trust Barometer, released in January, trust in U.S. business dropped from 58% to 38% in one year. European businesses are in nearly as much trouble with the public. Businesses in emerging markets are faring better—but not by a lot.

If companies can’t address this problem, an economic turnaround may be delayed indefinitely: Banks won’t lend money; innovation will slow to a crawl; trade across borders will fall even more rapidly; governments will overregulate the private sector; unemployment numbers will continue to rise; and consumers won’t open their wallets for anything they consider nonessential. A complex modern economy simply can’t function unless people believe that its institutions are fundamentally sound.

The articles' executive summaries are:

Rod Kramer, in “Rethinking Trust,” argues that most of us trust others far too easily. While the pundits claim that businesses need to rebuild consumers’ trust as soon as possible, Kramer argues that we need to remain skeptical.

Joel Podolny’s piece “The Buck Stops (and Starts) at Business School,” indicts the schools that train managers and executives and shares his thoughts about how to reinvent business education—and thereby regain people’s trust.

James O’Toole and Warren Bennis argue passionately that senior managers must build a culture of transparency to repair that problem. “What’s Needed Next: A Culture of Candor” makes the case that trust within organizations is the bedrock for rebuilding it in business as a whole.

In “How to Be a Good Boss in a Bad Economy” Bob Sutton and another Stanford professor take a different perspective on the destructive dynamic of senior management behavior during tough economic times — and describes a better one.

There is also an article from the previous month's issue about "When Contracts Destroy Trust."

Rewiring the Primal Management Talent Code

2009-05-26T07:10:00.005-05:00

I came across an interesting book in Borders over the weekend, but didn't have the time to browse it more thoroughly. A few hours later, at home, I looked it up on Amazon.com. I found the description and review comments very interesting, and found myself following links to some related books and reading through those pages as well.

There were three books in particular that struck me as having conclusions that were different, but closely connected, and which combined together to yield something more powerful than any of them does alone. These three books are:

Management Rewired: Why Feedback Doesn't Work and Other Surprising Lessons from the Latest Brain Science, by Charles S. Jacobs

Primal Management: Unraveling the Secrets of Human Nature to Drive High Performance, by Paul Herr

The Talent Code: Greatness Isn't Born. It's Grown. Here's How., by Daniel Coyle

What do you think? Can you see a connection from each of these to the other that suggests a "bigger picture" regarding agility, collaboration, leadership and excellence? How did you connect the dots from one to the other?

Synchronous and Staged Integration

2009-05-21T00:56:00.006-05:00

I participated in a LinkedIn CM group discussion about Building Code before -vs- after Checkin. The discussion was kicked-off by Tracy Ragan, COO and Co-Founder, OpenMake Software:

Many companies implementing a distributed SCM process make the mistake of checking source code into their SCM repository before they validate the code through a compile and link process. Checking in source code that does not compile is honestly, a waste of time. I call it the garbage in/garbage out method. The goal of SCM is to match your production source code to your production executables. This goal should be kept in mind when implementing your SCM process.

So many companies have a very complex SCM process with tightly managed approvals. But when it comes time to roll out binaries to production, they have no idea how those binaries were created. What you need is the ability to run a footprint of your production executables showing all artifacts used to create those binaries. That footprint should show the versions of the files that were found via your SCM repositories and audit all files that were used to create the binary but were not stored in your SCM repository.

Build your code as part of your SCM process. This is the only way to know if the code you are spending time and money to manage is actually executing in your production environment. The mainframe community has gotten this right for the last 20 years. It is time for the distributed developers to sort out a 100% complete SCM process.

There were several good comments, most of them tacking positions for or against, and a few adding some more insight. I responded as follows ...

I wrote a paper for the CM Journal on this very issue a few years back (Nov 2003). It was entitled Codeline Merging and Locking: Continuous Update and Two-Phased Commits

It talks about what we ideally want to have done by the time we try to commit our changes to the codeline (shared/team integration branch) and some of the different strategies (patterns) and trade-offs for how to ensure correct, complete & consistent results while still trying to be as practical as possible regarding complexity and overhead.

It does not however discuss the issue of "synchronous" versus "asynchronous" build+regression-test as part of the commit operation. It assumes "synchronous", where you must successfully build+test *before* the commit operation is considered complete (which is what Tracy is talking about here).

Another approach is "asynchronous", which is what many CI-server implementations do: allow the commit complete (perhaps after doing only an incremental build), but then behind the scenes immediately "kickoff" a more rigorous/complete build which then raises a visible alert upon failure (which should then be fixed *immediately*).

Rather than an either/or approach (building before commit -vs- building after commit), what is becoming more common for larger projects & codebases is a "staged continuous integration approach" such as those described by the following:

Multi-Stage Continuous Integration, by Damon Poole, December 2008, Dr. Dobbs Journal
Staged CI and the Build & Release Pipeline, by Maciej Zawadski, September 2008, Dr. Dobbs Journal
Is Pipelined Continuous Integration a Good Idea?, by Amr Elssamadisy, September 2007 from InfoQ.com
When to use Staged Integration, by Johanna Rothman, September 2007
Staged Builds with Cruise-Control, by Paul Duvall, April 2007
Staged CI Stories at Java Ranch Other references on Scaling Continuous Integration:
Measuring Continuous Integration Capability, Bas Vodde
Does Continuous Integration Scale to Enterprise Projects? [blog-article containing a whitepaper "PDF"]
Large-Scale Continuous Integration, by Damon Poole, CM Journal, January 2009, Vol. 8 No. 1
Avoiding Continuous Integration Build Breakage Patterns, by Slava Imeshev
CI Strategies: stage your builds
Staged Resilient Build Approach
Unbreakable Builds

The Five Traits of Agile Projects

2009-05-17T11:51:00.007-05:00

I submit that any project which successfully executes their practices in accordance with Agile values and principles will exhibit the following key traits (Note the acronym formed):

Adaptive -- responsive to change (based on feedback and learning), rather than predictive plan-driven. Plans, designs, and processes are regularly tuned and adjusted to adapt to changing needs and requirements
Goal/value-driven -- scope and solutions/activities are focused on and constrained by value-demand, producing executable-results (working functionality) in order of highest business value
Iterative & Incremental -- short development cycles, frequent releases, regular feedback
Lean -- simple design, streamlined processes, elimination of redundant information, and barely sufficient documentation and methodology
Emergent results from Evolvable structures -- successful results emerge over time, using close collaboration to create "firm yet flexible" design structures that are relatively easy to dynamically change & evolve into what the final result needs to be. This pertains to structures of team/organizations as much as it does to architecture, processes, requirements and plans.

No single one of these key traits is, by itself, unique to Agile, but I claim that all of them taken together form the differentiating characteristics of successful agile projects, teams and their environment. If they are visibly exhibiting these traits then the project and team are "agile."

Sensing and Sense-making in the Agility Cycle

2009-05-15T17:02:00.005-05:00

Part 1 of this series presented the Business Agility Cycle and from that derived the Software Agility Cycle in Part 2. Then part 3 elaborated upon the first step of that cycle, sensing the need for change using feedback-loops at all levels of scale.

In this installment (part 4) I intend to elaborate upon the second step of the software agility cycle: See the Problem in the Context of the "Whole" ...

Here the "whole" means both seeing and understanding the "big picture" and how any near-term/local changes we might make can impact or compromise the needs of everyone else in the value-chain. So how do we do this?

The basic way is to take a Systems Thinking approach within the (desired) context of a Learning Organization (particularly double-loop learning). That's a lot of fancy jargon without some very specific advice. There are a half a gazillion different ways to do that (see ValueBasedManagement.net for a bunch of 'em), so let's get a bit more concrete about some of the specific methods or means of doing this ...

In Lean terms, this would correspond to seeing the whole value-stream and optimizing the flow of operational value throughout the value-stream. Lean offers the technique of Value-stream mapping to help us see the problem/opportunity in the strategic context of the business.

From a Theory of Constraints (TOC) perspective, this would be akin to "the goal" of optimizing the throughput of value-delivery. TOC provides a whole host of Thinking Processes for us to figure out what to change, what to change to, and how to cause the change. The TOC Thinking processes are one specific set of tools for making strategic sense of the problem/opportunity with a focus on the end-goal.

There is also something called the Cynefin Framework (pronounced kun-ev’in) by Dave Snowden and his collaborators at IBM (see articles & resources at www.cognitive-edge.com). The Cynefin framework is a model used to describe problems, situations and systems. The model provides a taxonomy that guides what sort of explanations and/or solutions may apply. Basically you have to decide which of four categories your problem falls under: Simple, Complicated, Complex or Chaotic. There are pretty straightforward definitions for each of these in the framework so it's not too hard to take a stab at doing this. Then, each category has a recommended approach to use for strategizing:

Sense-Categorize-Respond to discover "best practice" (for Simple issues)
Sense-Analyze-Respond to discover "good practice" (for Complicated issues)
Probe-Sense-Respond to discover "emergent practice" (for Complex issues)
Act-Sense-Respond to discover "novel practice" (for Chaotic issues)

The above is oversimplifying a bit, so definitely visit the hyperlinks to find out more about this interesting framework.

What are some of the more concrete ways agile methods use to "keep their eyes on the prize" in terms of focusing on strategic value to the business and seieng the whole problem (so as not to "suboptimize")?

Using an onsite-customer (or customer-proxy) to maintain focus on business goals/priorities and always have the team working on the most highly valued features in the current iteration.
Applying the LRM principle or the YAGNI philosophy when tempted to anticipate a feature or add additional flexibility/capability to a design.
Applying the DRY principle and/or refactoring to make designs be simple and sufficient
Using TDD to specify a requirement as a test, and then writing just enough code to pass the test (and using Acceptance TDD at the story/feature-level)
Having the "navigator" in a pairing session focus the "driver" on strategic goals.

Some of these are more fine-grained than others, but they are valid examples of the approach used when responding to a problem/opportunity sensed by one of our feedback-loops. Agile methods use a validation-driven approach where, for a given activity, we have a definition of "DONE" that we can systematically verify through either automation or rapid stakeholder communication, and keep using that "DONE" criteria as the "carrot" to keep us aligned with our strategic goals, even at the tactical level.

BOOK: Agile Testing

2009-05-13T23:51:00.004-05:00

I reviewed Agile Testing: A Practical Guide for Testers and Agile Teams for the May 2009 issue of The Agile Journal

The book is exactly what its title says, and should quickly become “the bible” for all would-be agile testers. Right from the start it is obvious that this book is not about theory, but is borne from profoundly deep insight and broad experience on real world projects. It doesn’t just cover testing types and techniques, or values and principles, but also real world challenges from organizational culture to team logistics; and from technological and geographical constraints to tooling, environments, and communication/collaboration.
...

If your organization has dedicated testers and test-teams and wants (or needs) to learn how to work effectively on agile projects, or even just in an agile fashion, I currently cannot envision a better or more practical “HowTo” guide then Lisa Crispin’s and Janet Gregory’s Agile Testing: A Practical Guide for Testers and Agile Teams. I expect it to appear at the top of any mandatory reading list about learning and setting-up a discipline of agile testing.

You can read the full review here

BOOKS: The CMDB Imperative and the Data Access Handbook

2009-05-09T23:17:00.001-05:00

I recently received the following books:

The CMDB Imperative: How to Realize the Dream and Avoid the Nightmares, by Glenn O'Donnel and Carlos Casanova; Prentice Hall PTR; March 2009

The Data Access Handbook: Achieving Optimal Database Application Performance and Scalability, by John Goodson and Robert A. Steward; Prentice Hall PTR; March2009

The CMDB Imperative has online excerpts and "extras" available from its InformIT.com homepage. It also has its own website at cmdbimperative.com where you can find additional resources, previews and excerpts, and a cmdbimperative blog. The "blurb" for the book is:

Implement Configuration Management Databases that Deliver Rapid ROI and Sustained Business Value. Implementing an enterprise-wide Configuration Management Database (CMDB) is one of the most influential actions an IT organization can take to improve service delivery and bridge the gap between technology and the business. With a well-designed CMDB in place, companies are better positioned to manage and optimize IT infrastructure, applications, and services; automate more IT management tasks; and restrain burgeoning costs. Now, there’s an objective, vendor-independent guide to making a CMDB work in your organization. The CMDB Imperative presents a start-to-finish implementation methodology that works and describes how the CMDB is shifting to the superior Configuration Management System (CMS).

Expert CMDB industry analyst Glenn O’Donnell and leading-edge architect and practitioner Carlos Casanova first review the drivers behind a CMDB and the technical, economic, cultural, and political obstacles to success. Drawing on the experiences of hundreds of organizations, they present indispensable guidance on architecting and customizing CMDB solutions to your specific environment. They’ll guide you through planning, implementation, transitioning into production, day-to-day operation and maintenance, and much more. Coverage includes:
Defining the tasks and activities associated with configuration management
Understanding the CMDB’s role in ITIL and the relationship between CMDBs and ITIL v3’s CMS
Building software models that accurately represent each entity in your IT environment
Ensuring information accuracy via change management and automated discovery
Understanding the state of the CMDB market and selling the CMDB within your organization
Creating federated CMDB architectures that successfully balance autonomy with centralized control
Planning a deployment strategy that sets appropriate priorities and reflects a realistic view of your organization’s maturity
Integrating systems and leveraging established and emerging standards
Previewing the future of the CMDB/CMS and how it will be impacted by key trends such as virtualization, SOA, mobility, convergence, and “flexi-sourcing”

The Data Access Handbook also has online excerpts and "extras" available from its InformIT.com homepage as well as its own website at dataaccesshandbook.com where you can find additional resources, excerpts and code-samples and its own communiuty-site. The "blurb" for The Data Access Handbook is:

Drive breakthrough database application performance by optimizing middleware and connectivity. Performance and scalability are more critical than ever in today’s enterprise database applications, and traditional database tuning isn’t nearly enough to solve the performance problems you are likely to see in those applications. Nowadays, 75-95% of the time it takes to process a data request is typically spent in the database middleware. Today’s worst performance and scalability problems are generally caused by issues with networking, database drivers, the broader software/hardware environment, and inefficient coding of data requests. In The Data Access Handbook, two of the world’s leading experts on database access systematically address these issues, showing how to achieve remarkable improvements in performance of real-world database applications.

Drawing on their unsurpassed experience with every leading database system and database connectivity API, John Goodson and Rob Steward reveal the powerful ways middleware affects application performance and guide developers with designing and writing API code that will deliver superior performance in each leading environment. In addition to covering essential concepts and techniques that apply across database systems and APIs, they present many API examples for ODBC, JDBC, and ADO.NET as well as database system examples for DB2, Microsoft SQL Server, MySQL, Oracle, and Sybase. Coverage includes:
Clearly understanding how each component of database middleware can impact performance and scalability
Writing database applications to reduce network traffic, limit disk I/O, optimize application-to-driver interaction, and simplify queries—including examples for ODBC, JDBC, and ADO.NET
Managing connections, transactions, and SQL statement execution more efficiently
Making the most of connection and statement pooling
Writing good benchmarks to predict your application’s performance
Systematically resolving performance problems—including eight start-to-finish case-study examples

An Agile CM Manifesto: Lame Oxymoron or Long Overdue?

2009-05-07T00:02:00.002-05:00

I submitted a proposal on this topic to the Agile2009 conference. The idea was to garner feedback as to whether or not there is a perceived need for Lean/Agile CM Manifesto (or "Declaration" of some sorts), which sort of presumes there is a legitimate place for something called Agile CM (or Lean CM).

The proposal was well received by its reviewers, but alas the sheer number of submissions versus number of available slots meant that even a lot of well-received submissions (including mine) didn't make the final cut.

Here is the proposal (below)! What do you think about the basic question it would ask of its audience? Is there a legitimate need for a Lean/Agile CM Manifesto? If so, what do you think it should say?

Summary Description

Agile development, agile project management, agile management, agile testing, all thus far have grown sizeable communities founded by many respected experts in their field. Why is this not yet the case for agile configuration management? Is there simply no need? Is lean/agile CM an oxymoron? Or is it an idea whose time has come and is long overdue? This talk will explore common complaints and misunderstandings between agilists/developers and CM, define what lean/agile CM really means, and whether or not a corresponding “manifesto” for CM is warranted (and if so, what must it include).

Presentation Outline

Approx ~30min of presentation followed by discussion/dialogue with the audience on whether or not the world needs a Lean/Agile CM manifesto, and what it should say. The Outline follows:

What is CM? (its more than just integration/build and version-control)
Traditional CM definition and Lean/Agile perspectives on CM
What is “Agile CM”? (CM for Agile projects? Agility for CM? or both?)
Lean/Agile CM Planning?
Lean/Agile Change Control/Tracking?
Lean Configuration audits/reviews, and status accounting?
Lean Traceability? (everyone’s favorite)
Agile Version control and Lean branching
Agile integration & build (nested synchronization & harmonic cadences)
“Emergent CM Architecture” from “refactoring” to SCM patterns

Discussion Points:

Common agilist/developer complaints & misunderstanding about CM [interspersed with the presentation]
Common CM complaints about (agile) development [interspersed with the presentation]
Do we need a Lean/Agile CM manifesto? Why or why not? [at the end of the presentation]
What must this manifesto include? from whom? [at the end of the presentation]

Background/Materials:

Materials for the presentation will be distilled from the following sources where many of the points above have been presented or discussed in more detail. Each of the below will be distilled into no more than a single slide (with few exceptions):

For additional background, links to a veritable cornucopia of related articles may be found on the CMWiki-web at http://cmwiki.com/AgileSCMArticles

Learning Outcomes:

Learn what Lean/Agile CM really means & implies
Common misunderstandings of agilists and developers about CM, and vice-versa
How to apply Lean thinking and Agile principles to more than just CI (CM planning, change-tracking, version-control, etc.)
Discover why there is (or is not) a need for a Lean/Agile CM “manifesto” or “declaration of interdependence”

Dee Hock on Hiring & Leadership

2009-05-04T23:53:00.002-05:00

I came across a great quote from Dee Hock in an article at Good2work.com:

“Hire and promote first on the basis of integrity; second, motivation; third, capacity; fourth, understanding; fifth, knowledge; and last and least, experience. Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind. Experience is easy to provide and quickly put to good use by people with all the other qualities.”

A few other good quotes from Dee Hock ...

“If you don't understand that you work for your mislabeled 'subordinates,' then you know nothing of leadership. You know only tyranny.”

“If you look to lead, invest at least 40% of your time managing yourself - your ethics, character, principles, purpose, motivation, and conduct. Invest at least 30% managing those with authority over you, and 15% managing your peers.”

“If you're in such a position of power and your ego is such that this is not possible, then its essential to have a small cadre of very bright, committed people who are questioning, exploring and understanding these emerging concepts.”

“It is essential to employ, trust, and reward those whose perspective, ability, and judgment are radically different from yours. It is also rare, for it requires uncommon humility, tolerance, and wisdom.”

“Lead yourself, lead your superiors, lead your peers, and free your people to do the same. All else is trivia.”

“Make a careful list of all things done to you that you abhorred. Don't do them to others, ever. Make another list of things done for you that you loved. Do them for others, always.”

“Money motivates neither the best people, nor the best in people. It can move the body and influence the mind, but it cannot touch the heart or move the spirit; that is reserved for belief, principle, and morality.”

BOOK: Pragmatic Thinking and Learning

2009-05-02T01:11:00.003-05:00

The Pragmatic Bookshelf totally rocks!!! I received my 3rd edition of the pickaxe book: Programming Ruby 1.9, which reminds me that I'm anxiously awaiting the arrival of Programming Scala (some excerpts are available now).

The "Pragmatic" book I most recently finished is Pragmatic Thinking & Learning: Refactor Your Wetware, by Andy Hunt. Not quite two years ago, I tackled two similar books: Mind Hacks and Mind Performance Hacks. Both of those books cover a bit more material than PT&L does, but I found Pragmatic Thinking & Learning to be a bit more readable. It was also a good companion to the other two since I'd already gone through them once before. The coverage of R-mode versus L-mode reminded a bit of Dan Pink's A Whole New Mind: Why Right Brainers will Rule the Future.

So I'd have to say I definitely would recommend Pragmatic Thinking & Learning: Refactor Your Wetware, by Andy Hunt from the Pragmatic Programmer's bookshelf. Oh yeah! When I finished reading through the book, the 2nd-to-last page was a handy, dandy & concise summary of all the tips in the book. NICE!

The Agility Cycle - Part 3

2009-04-30T23:53:00.005-05:00

In Part 1 of this series we discussed the Business Agility Cycle and then in Part 2 we derived the Software Agility Cycle from that by applying "the people factor" of Agile development to the business-agility cycle.

That "people factor" of Agile development essentially boils down to the notion of emergent behavior/structure through self-organization of collaborative "agents." The resulting discussion used of lot of jargon from complexity science and wasn't particularly easy to follow. Feedback from one reader even suggested the resulting "steps" in the agility-cycle came across as so much Zen/Yoga mumbo-jumbo.

To make matters worse, I wasn't exactly ultra-consistent in how I characterized the cycle:

one description had six steps (sense, see, socialize, swarm, show, share)
another "condensed" the most closely-related steps together (sense + see, socialize + swarm, show + share)
then the summary appeared to have four steps (evaluate, collaborate, validate, learn), which looks suspiciously similar to the Shewhart cycle of Plan-Do-Check-Act (though to be fair the differences between the two are important in meaning despite being only slight in appearance)

So perhaps the "jury" is still out on authoritatively characterizing those steps in the software agility cycle. Perhaps "strategize" is still better than "see from the perspective of the whole", even though doing the latter is really a prerequisite for being able to do the former correctly. And perhaps "swarming" and "socializing" are too readily misunderstood by those who don't already "grok" the whole notion of "emergence through self-organization" (and maybe don't really care to either).

The basic idea remains the same though: being "agile" means minimizing the response-time to some event or "stimulus" while maximizing the overall value of that response (and hence the efficiency and effectiveness of our behavior). This implies two basic things:

We must have some regular means of becoming aware of the "significant" events in the first place, or else the "cycle" never even starts-up in the first place.

There are multiple such "cycles" going on, each of which forms a feedback-loop at its own level of "scale."

So how do we "sense and make sense of" these events that indicate the presence of a need/opportunity for change? The answer is feedback loops! We have to mindfully and intentionally set them up ourselves and make sure they happen regularly, and at the right frequency and level of scale.

This is in fact how the software-agility cycle fits into the larger business-agility cycle. If we think of the business-agility cycle(s) as something that takes place at the level of entire portfolios, product-lines, markets, programs and their governance, then ultimately:

The very software project/product we're trying to be "agile" for came about in response to some higher-level business-need.

And putting an "agile" project+team in place was really the "act" step of the business-agility cycle.

The act of putting that agile project into motion is what prompted us to set-up the feedback-loops for software agility for that product or service.

What then do these feedback-loops look like and how do we put them into place? Well, they typically need to validate our knowledge and understanding of a need/opportunity against that of the user or consumer in some systematic fashion. For software agility, these feedback-loops are manifested by some of the agile software development practices we have become quite familiar with:

Iterations:

Retrospectives

Stand-up Meetings:

Continuous Integration:

Test-Driven Development:

When done by the programmer at the unit-level, TDD forces the granularity of this feedback cycle to be pretty darn small (often just hours, or less).

At a higher-level of granularity is Acceptance Test Driven Development (ATDD) where customer-acceptance criteria are represented as readable yet "executable requirements" in the form of automated tests. (These, in turn, drive the TDD cycle at the unit-level.)

Pair Programming:

One picture that is particularly good at depicting several of these feedback-loops all working together is the agile development poster from VersionOne:

Unfortunately, in order to see the picture at larger-size, you'll need to request it from VersionOne (It is free, but you have to fill-in a web-form to obtain it.)

Every single one of the above practices establishes a systematic feedback-loop whose purpose to “sense” some form of problem/opportunity by comparing our current understanding of something and validating against that of the consumer.

Each loop progresses through the software-agility cycle at its own level-of-scale.
And each one of them requires being able to “make sense” of the problem/opportunity after you’ve sensed it, by “seeing the problem in the context of the whole”
This requires us to think strategically about the end-goal before adding that unneeded abstraction or anticipating that not yet high-priority requirement, or fixing that urgent build-breakage with a band-aid that actually makes things harder for the next “consumer” downstream in the process).

So if the "secret sauce" of software agility comes from the "people factor" to create emergent results from close collaboration, then the "secret recipe" for applying that sauce is the "nested feedback-loops" that integrate the collaboration and resulting "emergent behavior" into adaptive cycles of activity that let us incrementally learn and evolve our understanding by iterating through each of those cycles at each level of scale.

BOOK: Trust - The social virtues and the creation of prosperity

2009-04-29T23:41:00.009-05:00

A couple of posts ago I mentioned the book Trust: The social virtues and the creation of prosperity by Francis Fukuyama. I hadn't read the book yet, but Austin Hastings, an esteemed SCM colleague, has and I promised that if he was willing to post his summary and notes on the book that I would publish them on my blog here (and not just as a comment). I also recommend reading one of Austin's recent blog-entries entitled "Being a trust specialist", which reinforces why, even for CM professionals, it is often all too true that "The first thing to 'build' is TRUST." The rest of this blog-entry is Austin's writing.

Francis Fukuyama is noted for writing "The End of History" and "The Great Disruption" as well as for "Trust: The Social Virtues and The Creation of Prosperity." As such, it is pretty easy to draw inferences about his political viewpoint(s). Many reviewers have made the assertion that his books are a product of his politics. I (personally) don't know enough to comment.

That exact argument -- questionable causality -- is one they make against some of the correlations that Fukuyama cites in Trust, but they seem to miss it when it applies to their own positions. So beware: the book may be written in deliberate support of his politics, or his politics may have evolved from the studies he has done. It's your call.

Fukuyama claims that trust is a form of "Social Capital." That is, trust is something that you can invest in, something that you can create or obtain more of, something that you can use to achieve an economic end, and something that has value.

He further claims that the presence or absence of trust in a society has a significant, measurable impact on the economic indicators for that society, and that it probably has other, less clear effects as well. All of this is generally in agreement with other writings on trust that Brad has cited here.

His argument, then, is that there should be a way to objectively test for the amount of trust in a society. His test criterion is the number of employees that work for medium-sized businesses, where business size is a function of the number of employees. (That sounds circular, but it isn't.)

For example, my neighborhood pizza shop is run by George, his wife Rosanne, and their son Steve. There are two other cooks, and two delivery drivers. So the business has 7 employees, which makes it a small business.

By contrast, GM just announced that they intend to close down the Pontiac car brand, and lay off 21,000 workers. Having 21,000 workers to lay off makes GM a really, really HUGE business.

Somewhere between the two extremes lies a medium sized business. I don't recall exactly what numbers Fukuyama used, but let's say it's 100 <= n <= 1000. So we have small is less than 100 employees, medium is up to 1000, and large is anything over that.

With those boundaries in mind, the question is simply how many employees in a particular country work for a business that can be classified as "small," or "medium," or "large." Well, if there are 200 small businesses, and their average size is 30 employees, then 6,000 employees work for small businesses. (That's how we figured the average size, I guess.)

What Fukuyama found was that some regions or countries exhibited a noticeable "saddle shape" in their graph of business size versus number of employees. There were lots of people working for small businesses, and there were lots of people working for large businesses, but few people working for medium sized businesses.

He argues that large businesses are a distraction, because governments can create large businesses by fiat. France, Mexico, and most of the OPEC countries have huge businesses that were created by the government (as opposed to being grown up from small businesses).

Small businesses, naturally, are the starting point for almost everything. Somebody has an idea, they start a business with their close friend from college or their brother or parents, and if they make money they start hiring.

The problem comes when all the family members are hired. If the entire family is hired -- all the brothers, uncles, cousins, etc. -- and they're all doing some kind of management thing, with "outsiders" brought in to do the simple labor, the business has reached a critical point.

At this point, the business may or may not be *capable* of bringing in a qualified stranger, and handing that stranger an appropriate amount of power. That transition, from family shop to "real company," is the dividing line that Fukuyama is really looking for with his arbitrary criterion of 100 employees.

And he argues that if you see a disproportionately low number of workers that have jobs at medium-sized companies, it's because there are a low number of medium-sized companies. And *that* is because there is not enough social trust. When grandpa and dad and uncle Cletus can't let go of the reins, the company can't get any bigger. And in fact, the company will likely fail shortly thereafter, resulting in a much smaller business after the smoke clears.

Using his metric, there are some genuine surprises. Germany is a high-trust country, but France is low-trust. Southern Italy is virtually a no-trust area. Japan and Korea appear low-trust, until you refactor your statistics to deal with the Zaibatsu. China and most of the Asian mainland are low-trust.

This is interesting, but not necessarily controversial. What *is* controversial is the correlation with "The Protestant Ethic and the Spirit of Capitalism" (Max Weber, 1905 !!). That offends a lot of people, for a lot of reasons. Weber's point, made back when people were giving serious credence to "racial studies" and other stuff, was that Protestant countries did better economically than Catholic ones. Fukuyama makes a similar point, but claims that the effect is corollary, not causal.

Fukuyama's point is that there are a lot of flavors of protestantism, and countries that are majority protestant don't always have social mechanisms for creating and maintaining trust networks. If you can't generate trust, it doesn't matter how Protestant you are.

Things like the Rotary Club, and the Moose Lodge, and the Veterans of Foreign Wars in the United States do that job for us. These little mini-networks enable people of similar creed to reach each other, so that there are many networks of high trust -- "I trust him because we go to the same meetings" -- working parallel to each other. This enables Catholics to network with Catholics, Baptists with Baptists, Scrum fans with other Scrum fans, etc.

The rest of the "trust equation" is pretty straightforward. Nearly all of the trust literature agrees on these things: high trust leads to efficiency. Fukuyama's point is illustrated in any American business transaction. If you make noises of intent to engage in such a transaction, the expectation is that both sides intend to do so fairly.

That simple fact -- that you can go into a sandwich shop, for example, and place an order, and they will start making your order before you pay for it -- is one that is hard to see if you aren't looking for it.

The reverse situation pertains low trust areas. If you try to do business in these places, nothing is done until the cash changes hands, or at least until it is displayed for all to see.

A similar thing occurs in my own line of work, where implementing change tracking lifecycles is surrounded by requests to create explicit status codes for each possible situation. ("You should have a 'completed by development, but QA will not start testing due to other commitments' status code!")

It's important to keep in mind that the business-size metric is an indicator, nothing more. And that Fukuyama uses that metric as a way to set expectations for research, not as a causative for other social ills. A small number of medium-sized businesses doesn't cause poor social trust. It is an indicator that social trust is likely poor. (Or, as in the cases of Japan and Korea, that the metrics need to be refined.)

One of the reasons for many of the negative reviews is Fukuyama's assertion that trust is not correlated with equality, or fairness. American society has historically been more unfair and inequal than otherwise. Every single minority has been discriminated against at some point, which has led most of them to creating their own separate "civil associations" -- networks of trust.

The obvious inference is that eroding the separate networks of trust will result in an overall low-trust society. This isn't particularly politically correct, and so you can probably imagine how it was received in academic and/or liberal circles

Thanks again to Austin Hastings for taking the time to comment so comprehensively on this important work. And don't forget to read his recent blog-entry "Being a trust specialist."

BOOK: Building Trust

2009-04-25T00:12:00.003-05:00

Last time I blogged about Stephen Covey's The Speed of Trust: The One Thing That Changes Everything (see www.speedoftrust.com). I have a few other books about trust on my bookshelf, including:

Trust and Betrayal in the Workplace, by Dennis & Michelle Reina
The Trusted Leader, by Robert Galford and Anne Seibold Drapeau, and ...
Building Trust: in Business, Politics, Relationships, and Life by Robert C. Solomon and Fernando Flores

This time I'd I like to share some of the thoughts from Solomon and Flores' book Building Trust: in Business, Politics, Relationships, and Life. I bought it on the recommendation of David Anderson in his blog-entry You are what you read!. That blog-entry also mentions The Trusted Leader, as well as Trust: The social virtues and the creation of prosperity by Francis Fukuyama (which David felt was the most important book he'd seen on the topic so far).

Anyway, since we started exploring what trust is and what it means, here are some excerpt's from the introduction of "Building Trust: in Business, Politics, Relationships, and Life, by Robert C. Solomon and Fernando Flores (italicized comments appearing in square brackets outside of quotation marks are mine):

"Building trust begins with an honest understanding of trust, but it also requires everyday routines and practices. Without the practices, that understanding comes to nothing." [what happens if I replace "trust" with "agility" in this sentence]

"Trust is the essential precondition upon which all real success depends. The key to trust is action, and, in particular, commitment: commitments made and commitments honored."

"The problem of trust has clearly emerged as the problem in human relationships and organizations. What makes most companies falter-leaving aside market forces, bad products, and incompetent management-is the lack of trust."

"Trusting is something we make, we create, we build, we maintain, we sustain with our promise, our commitments, our emotions, and our sense of our own integrity. "

"Trust is not merely reliability, predictability, or what is sometimes understood as trustworthiness. It is always the relationship within which trust is based and which trust itself helps create."

"The freedom provided by trust is the freedom to to engage in projects that one could not or would not undertake on one's own. The freedom provided by trust is the freedom to approach and engage with strangers whom one may in fact never lay eyes on. The freedom provided by trust is the freedom to think for oneself and speak up with one's ideas. It includes as its consequence (not its cost) the freedom to be questioned and criticized -- and the right to be recognized and (if deserving) rewarded."

"Trust is a matter of making and keeping commitments, and the problem is the failure to cultivate commitment making.

"Trust involves sincerity, authenticity, integrity, virtue, and honor. It is a matter of conscientious integrity."

"Authentic trust is going into the unknown together."

"The worst enemies of trust are cynicism, selfishness, and a naïve conception of life in which one expects more than one is willing to give. Resentment, distrust, and inauthenticity are the result."

"Self-trust is the most basic and most often neglected from of trust. Distrust is often a projection of missing self-trust."

"Trust goes hand in hand with truth. Lying is always a breach of trust. What is wrong with lying, in turn, is that it breaches trust. ...telling the truth establishes trust and lying destroys it."

"Authentic trust can never be taken for granted, but must be continuously cultivated through commitments and truthfulness. True leadership, whatever else it may be, can be based on nothing less."

"cordial hypocrisy: the strong tendency of people in organizations, because of loyalty or fear, to pretend there is trust when there is none, being polite in the name of harmony when cynicism and distrust are active poisons, eating away at the very existence of organizations [or relationships]."

"How we think about trust ... makes trust possible, difficult, or even impossible. Trust (like love and freedom) involves any number of self-promoting and self-defeating prophecies."

"Trust(ing), not trustworthiness, is the issue. The existential question is how to trust, not just who can be trusted. (Trust is not only earned; it must be given.)

Trust is a matter of reciprocal relationships, not of predictions, risk and reliance.

Trust is transformative. It is not a matter of trusting or being trusted so much as it is a matter of changing each other and the relationship through trust."

"The German sociologist Niklas Luhmann stresses that trust is a way of dealing with complexity in an increasingly complex society. There is a deep truth to this. The paradigm of trust is not found in the simplicity of a familiar relationship. Rather, it exists in the new complexity of the world and the global economy. Trust not only lets us increase complexity in our lives (and thus simplify them at the same time); it also changes our lives in dramatic ways, allowing us to explore in new directions, to experiment and express ourselves in our relationships in ways that would otherwise be unthinkable. And it allows us to grow and change and mellow and deepen in all the ways that merely provincial trust and distrust distort and prohibit."

"Trust is not always a good thing. Trust can be foolish, naive, gullible, and blind. And trust ought never to be taken for granted. That is why we insist the issue is building trust -- that is, creating trust, maintaining trust, restoring trust once it has been lost or betrayed. We want to suggest that this requires a radical revision of our conception of trust. Our thesis, to put it simply, is that trusting is something that we individually do; it is something we make, we create, we build, we maintain, we sustain with our promises, our commitments, our emotions, and our sense of our own integrity.

Trust is not, contrary to what some authors have written, a medium, an atmosphere, a 'lubricant,' social 'glue,' a lucky break for one society or another, or some mysterious social 'stuff.' Trust is an option, a choice. It is an active part of our lives, not something that is there from the beginning, or that can be taken for granted. It involves skills and commitment, not just good luck or mutual understanding.

The focus of trust -- or what we will call authentic trust -- is not just the hoped for outcome of this or that event or transaction. Trust is not merely reliability, predictability, or what is sometimes understood as trustworthiness. It is always the relationship within which trust is based and which trust itself helps create. Authentic trust does not necessitate the exclusion of distrust. To the contrary, it embraces the possibilities of distrust and betrayal as an essential part of trust. To be somewhat grim in our initial characterization of trust, it entails the possibility of betrayal.

The loss of trust is not mere disappointment. That is why trust is often evident only in the event of a breakdown. Like love, trust often becomes most palpable in the breach. (“You don't miss your water till the well runs dry.”) Building trust means coming to terms with the possibility of breach and betrayal."

"Trust, like love, may seem to fail us, but truly, we fail at trust or love. But then we get more sophisticated. We learn that trust, like love, is an emotional skill. It requires judgment. It requires vigilant attention. It requires conscientious action. It involves all of the intricate reciprocities of a human relationship (even in cases in which it remains “unrequited”)."

"Trust. like love, is an emotional skill, an ongoing dynamic aspect of relationships. We don't just fall in love, we decide to love. So, too, we do not simply find ourselves trusting, after months or perhaps years of comfortable familiarity. We make decisions to trust. We make promises and tacit communication. We see them through. We come to have expectations of others, and we respond to the fulfillment or frustration of those expectations. Trust isn't something we 'have,' or a medium or an atmosphere withing which we operate. Trust is something we do, something we make. Our mutual choices of trust determine nothing less than the kinds of beings we are and the kinds of lives we will live together."

Note some of the similarities and differences between the above, and what Stephen R. Covey writes in The Speed of Trust. Next time I'll give a few more resources on trust!

BOOK: The Speed of Trust

2009-04-24T02:05:00.003-05:00

I read Stephen Covey's The Speed of Trust: The One Thing That Changes Everything (see www.speedoftrust.com). I listened to it on audiobook during my commute a few months ago and parts of it definitely struck a chord with me.

I like how he described the relationship between "trust" and speed+cost, and how low trust makes things slower and more costly. He also defined "5 levels of trust" (more like concentric circles) as follows:

Self-Trust ("giving trust" - do you trust yourself? are you willing & able to trust of others?)
Relationship Trust (establishing trust within interpersonal relationships)
Organizational Trust (establishing trust within & across an organization)
Market Trust (establishing trust within & across your market - stockholders, patrons, consumers. This is like "Brand trust")
Global Trust (I forget the examples of this one)

There is a good summary of the book here, and another one here. There is also an early draft of chapters 1-2 available online.

Covey actually doesnt try to define trust very precisely. He simply quotes Jack Welch saying "I know it when I feel it." He says trust implies confidence in something/someone, and that lack of trust implies suspicion.

Trusting someone is a function of our perception of their character, and their competency.

"Character" includes a person's integrity, motives/agenda, intent and behavior with people, where
"Integrity" is mostly congruence (with an appropriate dash of humility and courage thrown in). Lack of congruence results in a lack of credibility.
"Competency" includes a person's capabilities, skills, results and "track record"
"Capability" is defined in terms of TASKS: talent, attitudes, skills, knowledge & style.

In addition to observing the five "waves" (or "levels of evolutionary scale") of trust above, he says about the first level/wave that it is very much about credibility, and describes the "four cores" of credibility are a person's integrity, intentions, capability and results. Demonstrating those things builds credibility in your words and actions. Credibility is a necessary (but not sufficient) ingredient for trust. People are less able to trust you if they don't find you credible.

The rest of the book is about the so called "13 behaviors" that, when demonstrated, will help you "build trust". Those are:

talk straight
demonstrate respect
create transparency
right wrongs
show loyalty
deliver results
get better (improve)
confront reality
clarify expectations
practice accountability
listen first
keep commitments
extend trust

Regarding "higher" (more evolved) levels of trust, he refers to the following principles:

the principle of alignment
the principle of reputation
the principle of contribution

He also describes some myths about trust:

MYTH	REALITY
Trust is soft.	Trust is hard, real, and quantifiable. It measurably affects both speed and cost.
Trust is slow.	Nothing is as fast as the speed of trust.
Trust is built solely on integrity.	Trust is a function of both character (which includes integrity) and competence.
You either have trust or you don’t.	Trust can be both created and destroyed.
Once lost, trust cannot be restored.	Though difficult, in most cases lost trust can be restored
You can’t teach trust.	Trust can be effectively taught and learned, and it can become a leverageable, strategic advantage.
Trusting people is too risky.	Not trusting people is a greater risk.
You establish trust one person at a time.	Establishing trust with the one establishes trust with the many.

The Agility Cycle - Part 2

2009-04-17T17:19:00.015-05:00

We saw in the previous blog-entry several definitions of the Business Agility Cycle. We also mentioned that in order to derive the Software Agility Cycle from this, we needed to explicitly include more close collaboration.

The Software Agility Cycle is:
Sense the Problem/Opportunity
See the Problem in the Context of the "Whole"
Socialize the Goals and Constraints
Swarm the Solution
Show the Results
Share the Knowledge Learned

Here's how I derived the above... Once again, I'll refer to Jim Highsmith to represent the "people and collaboration" component of software agility.

In his book Adaptive Software Development, Highsmith compares software development to a complex adaptive system (CAS) and uses CAS with elements of chaos theory and complexity science to derive the critical importance of people and collaboration in software development. He does this using the concepts of intelligent agents, self-organization and emergence within a turbulent (ever-changing, complex and seemingly chaotic) environment:

When we treat people as "intelligent agents" who both cooperate and compete to get work done in a turbulent environment, the final outcome is not the result of the work from any particular individual or process.

The outcome instead emerges from the interaction between the collaborating individuals to produce a result that cannot be predicted from their individual behavior (it is non-linear, defying simple cause-and-effect reasoning).

The details of the collaboration (and the final solution) cannot be pre-ordained and then commanded-and-controlled

Instead, successful collaboration and creativity must be nurtured, and must trust and empower the individuals to direct and organize themselves together "in real-time", learning and adjusting as they go.

This is the phenomenon of emergent behavior from self-organization to achieve successful results and adapt to unpredictable circumstances.

The other problem with the business agility cycle is that it seems to presume that decision-making about what solution to attempt is done by a smaller, separate group of people than those who will implement and deliver the solution, and we merely need to communicate to them and have them act to execute the solution.

It's not clear whether this assumes knowledge or design of the solution "up front" with a "handover" to an implementation team, or whether it can mean that determining the solution needs to be just as collaborative as its implementation and must involve many of the same people, all working together at the same time.

The collaborative aspect of software agility demands that the solution emerges from those who must create and deliver it, and that they are empowered to make the decisions about what that solution is and how best to do it. Rather than having the decision made for them and simply "executed" by them, once the problem became known, the request or opportunity would be presented to them as a problem to be solved, together.

The goals and objectives would be socialized, along with the needs and constraints, and then those who must collaborate across the value-stream to devise and deliver a solution would get together to make it a reality. They would learn what they needed to know, show results to customers and stakeholders and then get feedback to try and learn and adapt.

This yields a slightly different cycle for software agility than the one we had for business agility:

Sense the Problem/Opportunity using whatever local feedback mechanisms have been put in place to become aware of it. This is the same as it was in the business agility cycle.

See the Problem in the Context of the "Whole" system. Rather than "strategizing" this is more like the "orient" step from Boyd, or the "interpret & evaluate" step from Oosterhout. This lets us properly frame the problem within the organization and its "delivery system."

Socialize the Goals and Constraints among all those who will work together to devise and deliver the result. This is similar to the communicate step from Gartner, but we didn't really decide what the solution should be. We decided what to do (the vision) at a high-level but not the details of "how." We haven't made a detailed plan or spec.

Swarm the Solution, collaborating intensely and self-organizing as needed across functions, organizational and geographical boundaries, etc. This combines the "act" and "decide" steps into a single, self-organizing group responsible for the result. We delegated the decision-making authority to them about what exactly to build and how to go about building it.

Show the Results, demonstrating the knowledge gained about the solution in "executable" form so we can get feedback as to its validity. Then finally ...

Share the Knowledge Learned with the rest of the organization so they can inspect, adapt, and move forward. And most importantly so they can remember how they learned what they now know.

The thing to remember is that this cycle happens at every level of scale in our delivery system. If we do this cycle once for a large problem, we have ourselves a waterfall. The above cycle applies to ALL the feedback loops that need to be put into place. There is the delivery-cycle, the release-cycle, the planning-cycle (the timebox), the development-cycle (backlog-item), the integration-cycle (e.g., TDD and CI), and the peer-review cycle (e.g., pair programming). They all go thru these same six steps in some form or another:

Sense & See (sense locally, see globally)
Socialize & Swarm (socialize the problem, swarm the solution)
Show & Share (show the results, share the knowledge)

These are the steps of the software agility cycle that we use to Evaluate change, Collaborate together, Validate the results, and then Iterate over the cycle again, applying our newly found knowledge to learn and adapt was we go.

The Agility Cycle - Part 1

2009-04-12T09:36:00.016-05:00

Continuing the "What is Agility?" series of posts ... we have looked at business agility and how it combines with the "people factor" from the agile manifesto to yield software development agility. So now that we know the meaning of agility, the next questions two I want to answer are "how do you do it?" and "what does it look like?"

In this posting I will attempt to generally answer the "how do you do it?" question. This is basically a question of process. What is the overall process for "swiftly sensing and rapidly responding to change & uncertainty in close collaboration with stakeholders to create simple, sustainable structures with sufficient flexibility to dynamically adapt & evolve business processes, products & plans."

This process is essentially an overall cycle. Something that you do, and then "rinse, lather & repeat." It is called the Agility Cycle. Let's look at a few different descriptions of this cycle (mostly from the realm of business agility) and then formulate our own for software development agility.

Gartner describes the business agility cycle as:

Sense the need for change in the environment (includes the proactive initiation of change)
Strategize the available options and develop alternatives
Decide which path to take and commit to the approach
Communicate internally and externally to everyone who needs to know
Act to produce results and follow-through efficiently

Gartner has a number of reports on agility (some of which are even freely available :). You can read about them all at this summary.

John Boyd describes it as an OODA loop:

Observe your environment (yourself; your threats and opportunities; the physical, mental, and ethical situation; and potential allies and opponents)
Orient yourself to grasp what it all means and identify patterns, approaches and likely outcomes
Decide upon a course of action and tactical plan
Act on the plan (execute while minding the terrain)

Another source describes it as:

Scan emerging trends and issues through gathering information and analysis
Sense opportunities to translate information into actionable solutions
Respond to opportunities and risks by being sufficiently flexible at tactical and strategic levels
Shape future environments through driving change.

In Sense, Respond and Learn, Marcel v Oosterhout describes the business-agility cycle as:

Monitor & Sense: monitor the business environment and sense critical market signals, events, and changing conditions
Interpret & Evaluate: interpret the information and evaluate if it is something to (re)act to or ignore.
Decide & Plan: decide which response is best and plan to implement it
Reconfigure & Adapt: reconfigure or adapt business, operations, or IT capabilities
Respond & Execute: respond by executing with new or adapted capabilities
Learn: Record the cycle, learn from previous cycles, share to improve future cycles
Manage: manage the sense-respond-learn cycle to support completing the process as rapidly and effectively as required

Notice that, as with the definitions of business agility we saw before, there is no explicit mention of the kind of close collaboration and self-organization that is deemed an essential component of software agility. Any attempt we make to derive the software agility cycle from the business agility cycle needs to take into consideration this close collaboration of the participants and emergence of the solution as a result (rather than knowing it all up front).

We'll discuss this next time when we try to describe the software agility cycle.

What is Agility? Part 2 - Software Agility

2009-04-10T16:14:00.012-05:00

In Part 1 of "What is Agility?" we looked at numerous definitions and descriptions of business agility.

The overwhelming majority of them had the following elements in common among their descriptions:

Agility means swiftly sensing and rapidly responding to change and uncertainty.
Demand for change may be internal or external to the organization/environment.
Responding to the change utilizes resources & staff that are both internal and external.
Changes can be unpredictable and unanticipated, as well as foreseeable and anticipated.
Unanticipated changes are typically handled with dynamic adaptation through short cycles of feedback and learning.
Anticipated changes are typically handled with simple yet flexible structures that are resilient when modified.
And of course all this is achieved with solutions that are effective, efficient, and economical to implement and deliver!

So what is missing from the above that is an essential characteristic of agile development as defined by the values and principles of the Agile Manifesto?

I think it is the people factor, and what Jim Highsmith calls "a focus on talent and skills of individuals and teams." Close collaboration across functions and other internal & external boundaries is the "secret sauce" that best leverages the talents & skills of individuals & teams. This is what enables the rapid learning which, when combined with those talents and skills, delivers the most innovative solutions and best results in the shortest time.

Put ALL of the above together, and we have a definition of software development agility that looks something like the following:

Software development agility is the ability to ...
swiftly sense and rapidly respond to change & uncertainty
in close collaboration with internal and external stakeholders
to create simple, sustainable structures with sufficient flexibility
to dynamically adapt & evolve business processes, products & plans
that realize results with efficiency in motion, economy of effort, energy in execution, and efficacy of impact!

And there you have it! Note that the last bullet above is really just a fancy phrasing of mandatory business jargon that ultimately means "Lean", but we'll explore that connection further in a later blog-entry.

BOOK: Clean Code - A Handbook of Software Craftsmanship

2009-04-08T00:38:00.003-05:00

My review of Robert Martin's (and the rest of the folks at ObjectMentor) book Clean Code: A Handbook of Software Craftsmanship was just published in the April 2009 issue of the Agile Journal.

Robert ("Uncle Bob") Martin and the folks at ObjectMentor have written a new book that should be required reading for all programmers! When it comes to writing clear and maintainable code, cleanliness is indeed next to godliness, and we should all follow the Boy Scouts' Rule whenever we write or modify any piece of code: leave the place cleaner than when you found it!
Robert C. Martin is one of the giants of object-oriented design, and a founding father of the agile manifesto and of agile software development. More recently, he helped kickstart the software craftsmanship movement, which gained much attention from his keynote at Agile2008 and subsequent talks on craftsmanship and ethics.
...
If you are at all serious about programming, then as soon as you finish reading this review you should get your hands on Clean Code as fast as you can! Get it. Read it. Learn it. Then live it!

Read the full review

Additional resources related to the book are:

Uncle Bob's article "What is Clean Code?" which is an online version of the first chapter of the book
PDF of an excerpt from the book, including the TOC, Foreword, Introduction, and Chapter 1
Clean Code Tip of the Week #1 and other Clean Code tips available online from the "Extras" tab of the InformIT site for the book

CRISP Builds

2009-04-07T11:25:00.001-05:00

Jeff Langr and Tim Ottinger have a pretty sweet and useful "gig" going on at The Agile In a Flash project, which will "result in a book and replenishable deck of at least 52 flash cards."

I made a suggestion to them late last week about Mike Clark's "CRISP" acronym for builds, and they turned it into a CRISP flashcard. Here is the comment I made that led to that result ...

Mike Clark gave a 2004 presentation on Pragmatic Project Automation that included a description of what he called the "CRISP" criteria for build:

Complete (recipe lists all ingredients)
Repeatable (version control time machine)
Informative (radiate valuable information)
Schedulable (complete and repeatable)
Portable (machine-independent)

There is a similar description in the 2007 presentation “All Builds are Good”, and a more detailed description in this 2007 CT-SPIN presentation on project automation:

Complete:
• Build from scratch and independently without human intervention.

Repeatable:
• Must be able to create exactly the same build at a later time.
• Store build scripts in source control.

Informative:
• "Detector of unexpected changes".
• Provide information on why a build failed.

Scheduled:
• Let the builds run automatically.

Portable:
• Build should be runnable from any system (same platform), not just that of the developer.
• For cross-platform software, it should build on all platforms.

I'm curious as to whether or not others have come across this acronym and if they like it and find it useful? What corrections, clarifications or additions do you think should be made to the acronym or its description?

What is Agility? Part 1 - Business Agility

2009-04-06T07:03:00.013-05:00

Kicking off my series of posts on the subject of "What is Agility?" is this first posting on Business Agility, and how agile methods got its name from those roots...

Some of you may already know that before Agile methods used the word "Agile", the term "Lightweight methods" was being generally used to describe them. It wasn't until the February 2001 gathering at Snowbird, Utah that they began using the term "Agile", and the "Agile manifesto" was born.

At that time, the term "Agile" was already in use to refer to "business agility" by those in the field of organizational change and learning. I know of at least a few people (Mike Beedle for example) who were there at Snowbird that knew this and even liked the term and used it from time-to-time before that now famous gathering.

So the then lightweight methods community intentionally adopted the terms "agile" and "agility" in this context. Let's explore our roots and see exactly what "Business Agility" means. I surfed the net for some time and gathered a number of definitions.

Business Agility is ...

“the ability to both create and respond to [anticipated and unanticipated] change in order to profit in a turbulent business environment.” —[James Highsmith and Highsmith2]

“the ability to adapt rapidly and cost efficiently in response to changes in the business environment.” —[Wikipedia]

“the ability to sense environmental change and to respond efficiently and effectively to it. Sensing the need for change also includes the proactive initiation of change.” —[Gartner1 and Gartner2]

“the capability to be flexible, responsive, adaptive, and show initiative in times of change and uncertainty”—[BNET]

“the ability of enterprises to cope with unpredictable changes, to survive unprecedented threats from the business environment, and to take advantage of changes as opportunities” —[Agile Business: The competitive weapon. Market research study in four business segments (2004)]

“the ability to swiftly change businesses and business processes beyond the normal level of flexibility in order to effectively deal with unpredictable external and internal changes.” —[Business Agility in the Netherlands: Research Study Public Sector (2005)]

“the capacity to gain competitive advantage by intelligently, rapidly and proactively seizing opportunities and responding to threats.” —S. Meredith and D. Francis (2000)

“the ability to cope with unexpected changes, survive unprecedented threats from the business environment, and take advantage of these changes as opportunities.” —Z. Zhang and H. Sharifi (2000)

“The ability to thrive in a continuously changing, unpredictable business environment.” —R. Dove (1999)

Those are some nutshell definitions. They are focused on the dynamic capability to swiftly sense and rapidly respond to change using a combination of adaptation (for unforeseen changes) and flexibility (for foreseeable changes).

Let's look at some more details from a few more sources ...

From Process Transformation for Reaching Agility: CIO Role

Business agility calls for quick decisions and action. Agility is the ability of a business system to sense environmental change and respond efficiently and effectively to that change. Any framework for agility must address issues that go well beyond selection of the latest technologies.
Business system's willingness to be agile, its understanding of its own business system building blocks and its enablers of agility, and its adherence to an "agility cycle" are just as important as the judicious use of agility-influencing technologies.
The following four fundamental capabilities enable a business system to increase agile performance across the agility cycle. They are essential to being able to allocate corporate activities to measurable categories:
- Awareness (the right information through data and event monitoring mean knowing what is going on)
- Flexibility (the right options by rule modeling and simulation render possible confronting expected change)
- Adaptability (the right reactions by rapid rule modification enable confronting unexpected change)
- Productivity (the right policies, procedures and operations through automation for executing well day to day).

From What Lessons can the Agile Community Learn from a Maverick Fighter Pilot? and Thriving on Chaos:

The key to thriving on chaos is a skill Boyd calls agility, which is “the ability to rapidly change one’s orientation—[or] worldview—in response to what is happening in the external world.”
In the context of battle, agility involves four distinctive activities: observe your environment (yourself; your opponent; the physical, mental, and moral situation; and potential allies and opponents), orient yourself to decide what it all means, reach some type of decision, and attempt to carry out the decision. Observe, orient, decide, and act.
It turns out that the most critical step in the agility cycle is the first one: observe. This is much more than a process of looking around. Rather, it is a relentless search for the truth about your situation. The idea is to “go out and get all the information you can by whatever means possible.” Why? Because you can never be sure beforehand which stray idea will prove essential.
In Boyd’s conception, the first step—observe—is the only input from outside yourself. For this reason, how well your orientation matches the real world is largely a function of how well you observe. You are looking for mismatches between your current worldview and the world as it actually is. “A general rule is that bad news is the only kind that will do you any good….
You must seek out and find data that doesn’t fit with your worldview and you must do this while there is still time.” Otherwise the world will change and you will find yourself disoriented. You will have lost the initiative, which is dangerous in any conflict. However, if you are able to get the facts right and are willing to orient yourself to them, then astute decisions and effective actions will follow.

From Towards Agile Government:

In practice, agility features the following four characteristics: short term frontline responsiveness, strategic adaptation, outcomes focus. preventing or reducing problems before they arise.
Agile organisations are ‘hyper strategic’, tackling challenges wrought by turbulent external environments, while also preparing for future changes that are not yet apparent. They move through an agility cycle, seeking out and interpreting information to inform short, medium and long term decision making and action. The agility cycle is a four-step process through which organisations:
- scan emerging trends and issues
- sense opportunities to translate information into actionable solutions
- respond to opportunities and risks
- shape future environments.

From Agility thru SOA (Master’s Thesis)

Business Agility is the ability to sense internal and external changes, as well as being able to swiftly adapt, in reaction to sensed changes, businesses and business processes beyond the normal (operational) levels of flexibility, effectively using internal and external resources, to effectively manage unpredictable external and internal changes.
Different types of agility:
- Product Agility – the ability to easily change between various products to address changing needs
- Process Agility – the ability to easily adapt the internal processes to address changing circumstances
- Market Agility – the ability to easily enter and change markets
- Network Agility – the ability to easily coordinate and cooperate with various partners, ranging from suppliers to customers

So what did we learn? Share your thoughts! (I'll share mine in my next blog-entry.)

What is Agility?

2009-04-03T12:52:00.004-05:00

I'm involved in an Agile adoption effort that has done of lot of communication as of late. And one of the things we always cover in any intro training is the question What is Agile Software Development?"

Our standard answer and presentation materials usually involves citing the text of the Agile Manifestoand either this definition by Scott Ambler and/or this comment by James Highsmith that "Ultimately, Agility is about embracing change rather than attempting to resist it, [and a] focus on talent and skills of individuals and teams."

I've noticed our responses attempt to describe what Agile Development is by describing what it looks like (Ambler's definition), or the Agile values (from the manifesto), or the emphasis on change and collaboration (Highsmith). What they don't do is describe what Agility is! The Highsmith comment comes closest, but none of them really describe what it means for something to have the ability called agility.

I think it is key to understand what Agility is, and why/how it works, in order to understand Agile software development. I think without that understanding, it can be much harder to understand and apply the principles of lean/agile development, and to "inspect and adapt" to improve the right things in the right direction.

Toward that end, I did a bit of research on a lot of different definitions of agility (beyond some of my previous blog entries like Nutshell definitions of Agile development, Agile development distilled, and Business Agility defined) and of various attributes of agility. And I plan to share them in some blog-entries for each of the following:
Business Agility
The Agility Cycle
Software Agility
Self-Organization
Collective Intelligence
Social Creativity
Swarm Behavior
Emergence
Collective Ownership
Change & Uncertainty
Simplicity, Sufficiency and Sustainability

So "stay tuned" to this blog over the next several days because I intend to post on this more than weekly.

I'm ba-a-ack!

2009-04-02T18:34:00.002-05:00

I must apologies to my readers of this blog (or at least whomever is left). I haven't blogged in 9 months. I became heavily immersed in an agile adoption effort that had just managed to succeed in getting the senior-most management fully bought-in to Agile.

I've had a LOT of ideas I've been wanting to blog, many of them queued up and waiting, but I kept having too many other higher-priority things on my plate. I finally have a bit more breathing room now and plan to "catch up". So expect a burst of activity over the next few months.

Gee Scrum Method Owners!

2009-02-21T00:18:00.000-06:00

[Personal note: I first wrote this down back in April 2009 a few months or so before the Agile 2009 Conference in Chicago. I later put it down here in my blog with the intent to publish it but forgot to take it out of "draft" mode.]

Sung to the tune of Gee, Officer Krupke! from West-Side Story -- Music: Leonard Bernstein, Lyrics: Stephen Sondheim (see video of the movie performance)

Background:
For about a year now, I believe there has been a greater than usual amount Agile-related blogging and mail-list discussion on the subject of the shortcomings of what is currently the most popular agile development method: Scrum. Some of them are legitimately about Scrum, others are really more about the kinds of problems one encounters introducing Agile methods without appropriate emphasis/execution of the technical practices (especially Refactoring and TDD).

Its seems much of it may have started around the same time that Alan Shalloway (and later others, including Ron Jeffries, below) were banned from the ScrumDevelopment list for “going meta” and discussing problems with and/or additions to Scrum itself instead of how to apply/use it. Alan subsequently created the Lean-Agile list for such discussions (and is also working on a book).

As of this writing, more recent examples include the following:

James Shore, The Decline and Fall of Agile
Two specific threads on the ExtremeProgramming list: What about Scrum + CI + Automated-Tests? and What if Code Smells were counted as …
Robert (”Uncle Bob”) Martin, presentation on Craftsmanship and Ethics, and a follow-up blog-entry on Quintessence: The fifth element for the Agile Manifesto
The thread from the Lean-Agile list on Controlling Say (was Definition and Scope of Scrum / PO Role)
Ron Jeffries, Context My Foot! and Quality-Speed Tradeoff — You’re kidding yourself
Jurgen Appelo, The Decline and Fall of Agilists

Cast of Characters: You will need individual actors portraying the following roles!

Alan Shalloway, www.netobjectives.com
Ron Jeffries, www.XProgramming.com
Peter Alfvin,
Brad Appleton, blog.bradapp.net
Uncle Bob (Robert C. Martin), www.objectmentor.com

NOTE that the people are real, but the dialogue is fictional, even tho it is derived from bits and pieces of actual messages and blog-posts.

Lyrics:

RON (spoken): Hey, you!

ALAN (spoken): Me, Officer Jeffries?

RON (spoken):
That’s Scrum Master Jeffries to you! Gimme one good reason for not draggin’ ya to the scrumdevelopment list to answer for blamin all yer troubles on Scrum. What’ll you say to Schwaber, ya punk!

ALAN (sings):
Dear kindly Scrum-Lord Schwaber, You gotta understand,
It's just our agile nature that gets us Scrum-list banned.
Our management is clueless, our backlog’s full of junk,
Golly Moses, that’s where Scrum has flunked!

ALAN with BRAD & PETER:
Gee Scrum-method owners, we're very upset;
Your method doesn’t scale and we’re in technical debt!
We ain't no extremists, Our motives are clean,
Deep down inside us there is Lean!

ALAN: There is Lean!

ALL:
There is Lean, there is Lean, There is untapped Lean,
Deep inside XP and Scrum there’s Lean.

RON (spoken -- imitating Scrum list-owner):
That's an off-topic discussion!

ALAN (spoken):
Lemme tell it to the world!

RON (“Scrum list-owner”):
Just tell it some place else!

ALAN (singing, while creating the Lean-Agile list):
Dear Lean-Agile list-readers, The Scrum-list is too closed.
Despite their Agile leaders, the truth must be exposed:
They didn't want discussion of what Scrum oughta add.
Leapin' lizards, that's why I'm so “rad!”

ALAN:
Hey Scrum-method owners, you're really deep-fried!
There’s plenty wrong with Scrum that needs Lean-thinking applied!

PETER: Those darn product-owners have gotta be curbed!
RON: They’re pathologic'ly disturbed!
BRAD: They’re disturbed?

ALL:
They're disturbed. They're disturbed, They're the most disturbed,
Like they're pathologic'ly disturbed.

ALAN (spoken, imitating Mary Poppendieck):
Hear ye! Hear ye! In the opinion of Lean-thinking, these POs act depraved on account of they ain't had a proper “frame.”

BRAD (spoken): Hey, they’re depraved on account of their dis-functional context!
RON (spoken): Lemme to introduce ‘em to my fully functional foot!

BRAD (singing, to Ron)
My PO is so dastard, da coach won’t hear my plea,
da team’s code ain’t refactored, ‘cuz PO won’t agree.
They add it to the backlog for later to address,
Goodness gracious, dat's why it’s a mess!

RON:
Please! “Bad product-owners!” that reason’s so lame!
You shouldn’t even ask them, (and you should be ashamed)
It’s really yer context that oughta be changed,
It’s economic’ly deranged!

BRAD: We’re deranged?

ALL:
We’re deranged! We’re deranged! We are so de-ranged,
We're so economic’ly deranged!

RON (spoken):
In my opinion, Scrum don't need to have its head roles re-thunk at all. Use of RASCI matrices is surely a management disease!

BRAD (spoken): Hey, I got a management disease!

ALL (spoken sarcastically): So go buy an agile management tool.

BRAD (singing to "Agile tool-vendor"):
Dear kindly Scrum-tool master, they say go use your tools.
They’ll save us from disaster, ‘cuz they’ll enforce da rules.
And for our retrospectives, they’ll generate nice charts
Glory Osky, that's why our code smarts!

UNCLE BOB (with RON):
Yikes! Stupid tool vendors, you've done it again.
This team don't need a tool, just index cards and a pen.
It ain't just enough to inspect-and-adapt;

UNCLE BOB: Value craftsmanship over crap!
BRAD: Ship 'er crap?

ALL:
Don’t ship crap! Don’t make crap! Craftsmanship over crap!
Craft clean code as if you give a crap.

UNCLE BOB (spoken):
We hereby declare this project’s code to be crap on account of its ignerantive and excremental development. Failure to write clean code is professionally irresponsible! TDD and refactoring are not optional!

RON: The problem is your context!
ALAN: The answer is go Lean!
BRAD: The problem is design debt!
BOB: The answer is code clean!
PETER: The problem is Scrum's growing!
ALAN: The answer is Scrum's grown!

ALL:
Scrum Lords we got troubles of our own!
To all unclean-coders We just have to yell,
“Your projects can’t be agile if your code-bases smell!”
Gee, Scrum method-owners, What are we to do!
Dear Scrum method-owners, SCRUM YOU!!

Acknowledgements:

I credit Ron Jeffries with my inspiration (obsession?) in crafting the above! Ron quoted a line from the West-Side Story song “Gee Officer Krupke!” in one of his email replies and I couldn’t get the song out of my head all that day and began piecing together the above (which I developed iteratively & incrementally using TDD and refactoring – of course :-).

Summer of Books

2008-07-03T01:27:00.003-05:00

I'm going on some long needed (and hard earned) vacation. I won't be blogging again for about one month (so this will likely be my only entry for July).

I've got a lot of REALLY GREAT and interesting books to try and catch up on. I hope to blog about them when I return. Here is what's on my summer reading list:

Theory U: Leading from the Future as it Emerges, by C. Otto Scharmer: I saw the executive summary and other chapters at www.TheoryU.com (also see www.dialogonleadership.org). This looks to be THE penultimate book on leadership for the Agile Organization. It doesn't even use the word "Agile" anywhere, but the values and principles from the book are so well aligned with Agile values and principles, it is positively uncanny.
Advanced Rails Recipes (by Mike Clark), and Deploying Rails Applications (by Ezra Zygmuntowicz, Bruce Tate, and Clinton Begin): The latest Ruby & Rails books from the Pragmatic Bookshelf.
Implementing SOA: Total Architecture in Practice, by Paul C. Brown: I was extremely impressed with Paul Brown's earlier book on Succeeding with SOA: Realizing Business Value through Total Architecture, and am looking forward to this follow-up work.
Eating the IT Elephant: Moving from Greenfield Development to Brownfield, by Richard Hopkins and Kevin Jenkins: I received a review copy of this. Don't exactly know what to make of it just yet - but it does seem intriguing
Emergent Design: The Evolutionary Nature of Professional Software Development, by Scott L. Bain: This books looks to be an excellent "one stop shop" for learning the theory and practice of Test-Driven Development, Refactoring, Simple Design, and Design Patterns, all as part of a single integrated and coherent method
Agile Adoption Patterns: A Roadmap to Organizational Success, by Amr Elssamadisy: Amr has been writing for the Agile Journal and InfoQ.com and I have been looking forward to seeing this book in hardcopy.
The Software Project Manager's Bridge to Agility, by Michelle Sliger and Stacia Broderick: I think this is going to be THE book for all traditional PMPs trying to make the transition to agile project management.
Changing Software Development: Learning to Become Agile, by Alan Kelly: This books seems to have an interesting take on the connection between Agile, Lean, and Learning Organizations.

I have a few other books too, but they are much shorter :-)
Have a great July everybody!!!

Assigning Code Ownership-Policy Ownership

2008-06-26T01:45:00.007-05:00

Jurgen Appelo has an interesting article on StickyMinds entitled "Code Ownership Re-Visited"

Jurgen prefers the term "artifact assignment" rather than "code ownership" and explains there are 4 methods of artifact assignment:

Local Artifact Assignment (LAA) delegates policy to subsystems and subsubsystems (etc.)
Authoritarian Artifact Assignment (AAA) assigns change/access-control access-control of of ALL related artifacts to a single individual "benevolent dictator" who approves/authorizes all changes and who may also assign individual change-tasks to developers
Collective Artifact Assignment (CAA) assigns the whole team (rather than any one person) as collectively accountable for all its artifacts
Individual Artifact Assignment (IAA)

He also provides a nice set of criteria to help decide which policy to use.

This seems very different from the 4 kinds of ownership I described in "Situational Code Ownership: Dynamically Balancing Individual -vs- Collective Ownership" where I define what amounts to Non-Ownership (which can sometimes be the result of Dictatorship), Individual Ownership, Stewardship, and Collective Ownership and show how each maps to a corresponding leadership-style of the Situational Leadership Model.

So what gives? What explains this difference?

I think Jurgen would probably consider Stewardship as a weak form of Individual ownership (many others would too, though I staunchly disagree for reasons elaborated in the aforementioned article).
Authoritarian Assignment would be akin to the form of Non-Ownership that results from Dictatorship (or "director-ship" to be more precise) where assignments are made per modification/tasks by a director (or "benevolent dictator")
I would argue that the first two methods Jurgen describes above aren't really artifact-assignment policies, but instead are assignment-ownership policies: They're not so much about making a decision among ownership-policies as they are about making a policy for ownership decisions. Rather than deciding "who should own which artifacts", they decide "who should own the decision" to make such artifact assignments.

In other words, the first two policies Jurgen defines are about decision-rights to assign modification-rights to owners, and not about the modification-rights (or ownership assignments) themselves. As such, it raises an important point taken for granted in my article and in so many other discussions on this topic. Most of the prior discussion probably has assumed that the decision about which ownership-policy to adopt was made either "by the team" or by the team's "leadership" (that might be a manager, a technical-lead, a project-lead, or any combination thereof).

Another common assumption is that such ownership is defined along "architectural" boundaries such as individual artifacts/files, classes/modules, or packages, components and subsystems. Other possibilities are:

Functional boundaries (e.g., by feature/feature-set, story/theme/epoch, or use-cases)
Project boundaries (e.g., work-breakdown-structures, tasks, activities)
Role/discipline specialization boundaries (e.g., requirements, tests, models, database, UI, programming-language, user/help documentation, etc.), and even ...
Configuration/variation boundaries (e.g., version, variant, branch, platform). In fact some of these stretch across multiple dimensions of a project/product and might even be used in combination.

With Agile development, the emphasis is to break-down communication boundaries and any corresponding separation related to role, or phase, or physical boundaries and to instead prefer customer-determined boundaries of "scope" and "deliverable value" (e.g., stories, features or MMFs, use-cases, etc.).

So you will see a definite (but time-constrained) assigning of things like tasks to owners and stories (though the "owners" sign-up rather than "being assigned"). Those kinds of boundaries encourage closer and more frequent communication rather than separate & isolated (and less frequent) communication.

In the end, with Agile methods, it's all about maximizing learning and knowledge sharing & transfer rather than compartmentalizing knowledge into pigeon-holed roles and responsibilities. One opts for "separation of concerns" without "separation of those concerned" (work is isolated and separated, but not people).

Four Rules for Simple Codelines

2008-06-19T00:01:00.014-05:00

Some of you may be aware of Kent Beck's Four Rules of Simple Code that state simple code:

Correctly runs (and passes) all the tests
Contains no duplication (OnceAndOnlyOnce and The DRY Principle)
Clearly expresses all the ideas/intentions we needed to express (reveals all intent and intends all it reveals)
Minimizes the number of classes and methods (has no superfluous parts)

(I've seen some boil this down into some of the same rules for writing clear prose: correct, consistent, clear, and concise.)

Lately I've been noticing some parallels to the above and rules for what I would call "simple codelines" and I think there may be a similar way of expressing them...

Simple codelines:

Correctly build, run (and pass) all the tests
Contain no duplicate work/work-products
Transparently contain all the changes we needed to make (and none of the ones we didn't)
Minimize the number and length of sub-branches and unsynchronized work/changes

To elaborate further...

Correctly build, run (and pass) all the tests

This is of course the most obvious and basic of necessities for any codeline. If the codeline (or the "build") is broken, then integration is basically blocked, and starting new work/changes for the codeline is hindered.

Contains no duplicate work/products

The same work and work-products should be done OnceAndOnlyOnce! Sometimes effort is spent more than once to introduce the same change/functionality. This is sometimes because of miscoordination, or simply lack of realization that what two different developers were working on required each of them doing some of the same things (and perhaps should have been accomplished in smaller chunks).

Other times, rather than modify or refactor a common file, some will simply copy-and-paste the contents of one or more files (or directories/folders) because they don't want to have to worry about reconciling what would otherwise be merges of concurrent changes to the common files.

This is akin to neglecting to refactor at the "physical" level (of files and folders) as opposed to the "logical" level of classes and methods. It adds more complexity and (over time) inconsistency to the set of artifacts and versions that make up the codeline, and also eventually adds to the time it takes to merge, build, and test any integrated changes.

If content is being added to the codeline, we want that content to have to be added only once, without any duplicate or redundant human effort.

Transparently contains all the changes we needed to make (and none of the ones we didn't)

The above is sometimes the cause of much undesirable additional effort that is imposed for the sake of attaining traceability and ensuring process compliance/enforcement. Here, I mean to focus on the ends rather than the means, and I say transparency rather than traceability for that very reason.

If people are working in a task-based and test-driven manner, it should be simple to report what changes have been made since a previous commit and that only intended tasks were worked-on and integrated.

If a codeline is truly simple, then it should be very simple and easy to reveal all the changes that went into it without adding a lot of overhead and constraints to development. It should be easy to tell which changes/tasks have been integrated and what functionality and tests they correspond to. One very simple and basic means of tying checkins (or "commits") to backlog-tasks and their tests can be found here; others are mentioned in this article.

Minimizes the number and length of sub-branches and unsynchronized work/changes

Branching can be a boon when used properly and miserly. It can also add a heck of a lot of complexity and redundancy for maintaining two or more evolving variants of the project. The additional effort to track and merge and build many of the same fixes and enhancements in multiple configurations can be staggering.

Sometimes such branches are useful or even necessary (and can help with what Lean calls nested synchronization and harmonic cadences). But they should be as few and as short-lived as possible, preferably living no longer than the time it takes to complete a fine-grained task or to integrate several fine-grained tasks.

Even when there are no sub-codelines of a branch, there can still be un-integrated (unsynchronized) work-in-progress in the form of long-lived or large-grained tasks with changes that have not yet been checked-in or synced-up with the codeline. Keeping tasks short-lived and fine-grained (e.g., on the order of minutes & hours instead of hours & days) helps ensure the codeline is continuously integrated and synchronized with all the work that is taking place.

Another (possibly less obvious form) of unsynchronized work is when there is a discrepancy between the latest version of code checked-in to the codeline, and the latest version of code that constitutes the "last good build." Developer's lives are "simpler" when the latest version of the codeline (the "tip") is the version they need to use to base new work off of, and to update their existing workspace (a.k.a. "sandbox").

When the latest "good" version of the codeline is not the same (less recent) than the latest version, it can be less obvious to developers which version to use and become less likely that they use/select it correctly. Some use "floating tags" or "floating labels" for this purpose where they "move" the LAST_GOOD_BUILD tag from its previous set of versions to the current set of versions for a newly passed/promoted build. Sometimes the developers always use this "tag" and never use the "tip" (except when they have to merge their changes to the codeline of course).

Even with floating tags however, it is still simpler and more desirable when the last good version IS the latest version. Even if the latest version is known to be "broken", the lag between "latest" and "last good" version of a codeline can be a source of waste and complexity in the effort required to build, verify and promote a version to be "good" (and can introduce more complexity when having to merge to "latest" if your work has only been synchronized with "last good").

Plus, this lag-time often leads many a development shop to separate merging (and integration & test) responsibilities between development and so called integrators/build-meisters, where the best developers can attempt is to sync-up their work with the "last good build" and then "submit" that work to a manually initiated build rather than being directly responsible for ensuring the task is "done done" by being fully integrated and passing all its tests.

Such separation often leads to territorial disputes between roles and build/merge responsibilities. This in turn often leads to adversarial (rather than cooperative and collaborative) relationships and isolated, compartmentalized (rather than shared) knowledge for the execution and success of those responsibilities.

So there we have it! Four rules of simple codelines.

Simple Codelines should:
Correctly build, run (and pass) all the tests
Contain no duplicate work/work-products
Transparently contain all the changes we needed to make (and none of the ones we didn't)
Minimize the number and length of sub-branches and unsynchronized work/changes

Sometimes there are legitimate reasons why some of the rules need to be bent, and there are important SCM patterns to know about in order to do it successfully. But any time you do that, it makes your codeline less simple. So you want those scenarios to be few and far between, and to keep striving for the goal of simplicity. (Other SCM patterns, such as Mainline, can help you refactor your codelines/branches to be more simple.)

Traceability Matrix in an Agile Project

2008-06-12T00:57:00.002-05:00

InfoQ.com summarized an email-list discussion thread on the subject of using a Traceability Matrix in an Agile Project.

I contributed quite a lot to the thread, and InfoQ apparently included many of the key things I said along with the related URLs to articles I've written. (Thanks guys!)

Iterative and Incremental redefined redux

2008-06-08T00:06:00.007-05:00

The agile community has written much about this in the past year or so:

Iterative vs Incremental - from the first (and original) Wiki Web
The Neglected Practice of Iteratation - by Jeff Patton
Difference between Iterative and Incremental Development - nice visual depiction
No Waterfall Trap -- an even better visual depiction of the difference between iterative and incremental
Alistair Cockburn (pronounced "Coh-burn") has written several papers on the subject:
- Using Both Incremental and Iterative Development -- Crosstalk, May 2008
- Iterative and Incremental Development: How & Why you should be doing BOTH -- Better Software, April 2008
- Incremental means adding, Iterative means reworking -- Jan 2008 (I dont care for this definition, but the article has lots of good references/resources)
Iterative and Incremental Development: A Brief History -- by Craig Larman and Victor Basili, IEEE Computer, June 2003

Apologies in advance for being a "stick in the mud" on this one - I'm not particularly happy with the definitions so far. I searched around some more on the WWW and came across one I like a lot that I think better meets our needs.

It is from the paper What is Iterative Development? (part 1), by Ian Spence and Kurt Bittner,

Iterative and Incremental Development:
A style of development that involves the iterative application of a set of activities to evaluate a set of assertions, resolve a set of risks, accomplish a set of development objectives, and incrementally produce and refine an effective solution:
It is iterative in that it involves the successive refinement of the understanding of the problem, the solution's definition, and the solution's implementation by the repetitive application of the core development activities.²

It is incremental in that each pass through the iterative cycle grows the understanding of the problem and the capability offered by the solution.

Several or more applications of the iterative cycle are sequentially arranged to compose a project.
Sadly, development can be iterative without being incremental. For example, the activities can be applied over and over again in an iterative fashion without growing the understanding of the problem or the extent of the solution, in effect leaving the project where it was before the iteration started.

It can also be incremental without being truly iterative. For example, the development of a large solution can be broken up into a number of increments without the repetitive application of the core development activities.

To be truly effective the development must be both iterative and incremental. The need for iterative and incremental development arises out of the need to predictably deliver results in an uncertain world. Since we cannot wish the uncertainty away, we need a technique to master it. Iterative and incremental development provides us with a technique that enables us to master this uncertainty, or at least to systematically bring it sufficiently under control to achieve our desired results.

I like that this definition separated iterative from incremental and then defines them together. I would summarize it as follows (but I like the above better, even if it is longer):

Iterative development is the cyclical process of repeating a set of development activities to progressively elaborate and refine a complete solution. The “unit” of iterative development is an “iteration”, which represents one complete cycle through the set of activities.

Incremental development is the process of developing and integrating the parts of a system in multiple stages, where each stage implements a working, executable subset of the final system. The “unit” of incremental development is an “increment”, which represents the executable subset of the system resulting from a particular stage

Iterative and Incremental development is therefore ...

the application of an iterative development lifecycle to successively develop and refine working, executable subsets (increments) of a solution that evolves incrementally (from iteration to iteration) into the final product.
Each iteration successively elaborates and refines the understanding of the problem, and of the solution's definition & implementation by learning and adapting to feedback from the previous iterations of the core development lifecycle (analysis, design, implementation & test).
Each increment successively elaborates and refines the capability offered by the solution in the form of tangible working results that can be demonstrated to stakeholders for evaluation.

An Agile Iteration is a planned, time-boxed interval (typically measured in weeks) whose output is a working result that can be demonstrated to stakeholders:

Agile Iterations focus the whole team on collaborating and communicating effectively for the purpose of rapidly delivering incremental value to stakeholders in a predictable fashion.
After each iteration, the resulting feedback and data can be examined, and project scope & priorities can be re-evaluated to adapt the project's overall performance and optimize its return-on-investment

So in addition to the non-agile-specific definitions above, we see that Agile iterations are adaptive, in that they use the previous results and feedback to learn, adjust and recalibrate for the next iteration. And Agile increments are tangible, in that they can be executed and made accessible to stakeholders for demonstration and evaluation.

That's my story and I'm sticking to it!

The Laws of Codeline (Thermo)Dynamics

2008-06-02T02:10:00.002-05:00

Some of the discussion with my co-authors on our May 2008 CM Journal article on Agile Release Management spurred some additional thoughts by me that I hope to refine and work into a subsequent article later this year.

Release Management is about so much more than just the code/codeline (and it being "shippable") it's not even funny. Some other articles to reference and mention some key points from are:

Kevin Lee has written some GREAT stuff on Release Management that relates to Agile. The best is from the first and last chapters of his book on "The Java™ Developer's Guide to Accelerating and Automating the Build Process" but bits of pieces of it can also be found at:

ANY discussion about Release Management also needs to acknowledge that there is no single "the codeline", not just because I may have different codelines (Development-Line plus Release-Line) working toward the same product-release, but ESPECIALLY because no matter how Agile you are, the reality is that you will typically need to support MULTIPLE releases at the same time (at the very least the latest released version and the current under development version, but often even Agile projects need to support more than one release in the field)

So, when dealing with multiple release-line, and any "active development lines" for each of those, and the overall mainline, we really should say something overall about how to manage this "big picture" of all these codelines across multiple releases and iterations:

What is the relationship between development line, release-line and release-prep codeline?
How do the above three "lines" relate to "mainline"
What is the relationship between the different release-lines for the different supported releases
What is the overall relationship between the mainline and the release-lines (and if the mainline is also a release-line, which release is it?)

The above questions and the ability to give some big picture "advice" on relating it all together (the stuff of pattern languages) is precisely where Laura Wingerd's writing on "channeling the flow of change" and her chapter on "How Software Evolves" fits in! It tells us

The overall Mainline model
The different types of codelines ("line" patterns), and what kinds of builds take place on each of them
The relationships of those to the mainline
When+Why to branch (and from which "types" of codelines)
When+Why to merge across codelines (as a general rule)

These are where Laura's rules for "the flow of change" apply. And her concept of "change flow" is very much applicable to the Lean/Agile concept of "flow of value". The Tofu scale and "change flow" rules/protocol have to do with order+flow of codeline policies across the entire branching structure when it comes to making decisions about stability -vs- speed. One codeline's policy might make a certain tradeoff, but it is the presence of multiple codelines and how they work together, and how their policies define the overall flow of change across codelines, that forms the "putting it all together" advice that is key to release management across multiple releases+codelines.

In some way's you could make an overall analogy to the Laws or Thermodynamics and the realities of codeline management. Software and codelines tend, over time, to grow more complex and, if unchecked,
"Entropy" (instability) quickly becomes the most dominating force to contend with in their maintenance. See

"The 8 'Laws' of Software Evolution", and ...
"The Laws of Software Complexity"

The "entropy" (instability) doesnt just happen within a codeline. It can actually get far more hideous when it happens across codelines via indiscriminate branching from, or merging to, other codelines. This is what happens when you don't respect the principles and rules of "change flow" (from Wingerd) which ultimately stem from the rules of smooth and steady (value-stream) flow from Lean.

The Laws of Thermodynamics are about energy, entropy, and enthalpy. In the case of release management and codelines ...

energy relates to effort & productivity
entropy relates to stability/quality versus complexity
enthalpy relates to "order" (i.e., in the sense of structure and architecture as Christopher Alexander uses the term "order"). It is the "inverse" of entropy.

We could call them them "Laws of Codeline Dynamics" :-)

Energy misspent degrades flow, creates waste, and hurts productivity/velocity. In traditional development, we often see "fixed scope" with resources and schedule having to vary in order to meet the "scope" constraint. IN Agile development we deliberately "flip" that triangle upside down (see the picture in the article at here under the title "The Biggest Change: Scope versus Schedule - Schedule Wins"). So we are fixing "resources" and "schedule" and allowing scope to vary.

This might be one way of viewing the law of conservation of energy. If we fix resources and time (and insist on "sustainable pace" or "40hr work week") then we're basically putting in the same amount of effort over that time-box, but the key difference is how much of that effort results in "giving off energy" in the form of waste ("heat" or "friction") versus how much of that energy directly adds value. Both "Value" and "Enthalpy" degrade or depreciate over time, and adding more energy (effort) doesnt necessarily mean value is increased.

To make sure that energy goes toward adding value (and minimizing waste) we need to focus on the flow of value, and hence the flow of change/efforts to create value (the latter is one reasonable definition of a "codeline" or a "workstream"). to ensure a smooth, steady, and regular/frequent flow, there are certain rules we need to impose and regulate stability within and across codelines to better manage all those releases.

Zeroth Law of Thermodynamics (from Wikipedia)

- If two thermodynamic systems are each in thermal equilibrium with a third, then they are in thermal equilibrium with each other.

Translation to codelines ... this law of "thermal equilibrium" is a law of "codeline equilibrium" of sorts. (Does this mean If two codelines are are "in equlibrium" with a third codeline, then they are "in sync"? and with each other? here "in sync" doesnt mean they have the same frequency, it means their is some synchronization pattern regarding their relative stability and velocity. In Lean Terms, this would refer to "nested synchronization" and "harmonic cadence"). This might imply the "mainline" rule/pattern or one of Wingerd's rules of change-flow.

First Law of Thermodynamics

- In any process, the total energy of the universe remains the same.

This is the statement of conservation of energy for a thermodynamic system. It refers to the two ways that a closed system transfers energy to and from its surroundings - by the process of heating (or cooling) and the process of mechanical work.

This relates to effort & changes expended resulting in the creation of value and/or the creation of waste. We have activities that add value (which we hope is development), activities that preserve value (which is what much of SCM attempts do, given that it doesnt directly create the changes, but tries to ensure that changes happen and are built/integrated with minimal loss of energy/productivity/quality), and then we have activities (or portions of activities) that create waste (and increase entropy rather than preserving or increasing enthalpy/order)

Second Law of Thermodynamics

- In any isolated system that is not in equilibrium, entropy will increase over time

So this is the law of increasing instability/complexity/disorder. The "key" to preventing this from happening is achieving and then maintaining/preserving "equilibrium". How do we achieve such equlibrium? we do it with the "release enabler" patterns for codeline management (which help ensure "nested synchronization" and "harmonic cadence" in addition to achieving a balance or equilibrium between stability and velocity (to smooth out flow).

Third Law of Thermodynamics

- As temperature approaches absolute zero, the entropy of a system
approaches a constant minimum.

In our case, "Temperature" could be regarded as a measure of "energy" or "activity". As the energy/activity of a codeline approaches zero (such as a release in the field that youve been supporting and would LOVE to be able to retire that codeline sometime real soon), it's instability approaches a constant minimum.

This is perhaps another more polite way of saying something we already said in our article on "The Unchangeable Rules of Software Change", namely that "absolute stability" means dead (as in, "no activity"), and should serve as a reminder that our goals is not the prevention of change in order to achieve some ideal "absolute stability", for such an absolute would mean the project not just "done" but "dead".

On the other hand, it also speaks to us as a guideline for when it is safe to retire old codelines, and when to change their policy in accordance with their "energy level"

An Agile Approach to Release Management

2008-05-26T09:04:00.002-05:00

My Agile SCM co-authors Rob Cowham, Steve Berczuk, and myself have written an article for the May CM Journal on An Agile Approach to Release Management

We're relatively pleased with the article, and all collaborated together quite well.

BOOK: Software Teamwork - Taking Ownership for Success

2008-05-19T01:54:00.002-05:00

My review of Jim Brosseau's Software Teamwork: Taking Ownership for Success is available in the May issue of the Agile Journal. It is nothing less than outstanding!

I found Software Teamwork to be an immensely helpful, intensely practical, profusely insightful field guide to improving team outcomes and changing team behaviors by focusing on interpersonal action and personal leadership. This book belongs on any software team-leader's bookshelf, along with Jean Tabaka's Collaboration Explained and Murray Cantor's Software Leadership.

Other articles in this issue on the theme of "Challenges with Distributed Agile" are:

Distributed Version-Control Guide on InfoQ.com

2008-05-13T00:47:00.000-05:00

Nice little guide on InfoQ.com about Distributed Version Control - that's twice in two months that the "agile" section of InfoQ.com has had a decent article on the subject!

From PMBoK to Agility

2008-05-06T01:39:00.003-05:00

I recently learned that Michelle Sliger, author of the wonderful 4 part series of articles on Relating PMBoK to Agile Practices, is co-authoring a book with Stacia Broderick entitled the Software Project Manager's Bridge to Agility. You can even download an excerpt from her website.

I'm looking forward to this book a great deal, judging by the excellent articles and presentations of hers that I've read.

BOOK: Implementing ITIL Configuration Management

2008-04-30T00:59:00.002-05:00

I started reading through the book Implementing ITIL Configuration Management, by Larry Klosterboer. I'm really not what I'd consider an expert on ITIL nor IT Service Management, but I've had more than my fair share of exposure to it and am certainly no "slouch" in that area either.

This book looks to be an overview of ITIL and to how it applies to configuration management. From there one can extrapolate how much of it relates to CM for not just IT assets and infrastructure but to the software development environment and to software development itself.

The book includes coverage of the following (from the back cover):

Assessing your current configuration management maturity and setting goals for improvement
Gathering and managing requirements to align ITIL with organizational needs
Describing the schema of your configuration management database (CMDB)
Identifying, capturing, and organizing configuration data
Choosing the best tools for your requirements
Integrating data and processes to create a unified logical CMDB and configuration management service
Implementing pilot projects to demonstrate the value of configuration management and to test your planning
Moving from a pilot to wide-scale enterprise deployment
Defining roles for deployment and ongoing staffing
Leveraging configuration management information: Reporting and beyond
Measuring and improving CMDB data accuracy

To take the next step, and for a REALLY thorough treatment of how IT service management and CM comes full circle to embrace all of enterprise architecture and software development, I highly recommend Charles Betz' book Architecture and Patterns for IT Service Management, Resource Planning, and Governance: Making Shoes for the Cobbler's Children. As I mentioned in a blog-entry early last year, this book "really ought to be required reading for anyone that fancies themselves a 'CM professional' (especially Software CM) or an 'Enterprise Architect.'"

Three Pivotal Practices to Eliminate Waste

2008-04-22T01:26:00.000-05:00

I received my program for the Better Software Conference & Expo this coming June 9-12 in Las Vegas (alas, I will be unable to attend). The description for the keynote that will be given by Jean Tabaka caught my eye. Jean Tabaka is an Agile Coach from Rally Software Development and the author of Collaboration Explained: Facilitation Skills for Software Project Leaders).

Her keynote is entitled "Attacking Waste in Software: Three Practices We Must Embrace Now" and the abstract is as follows:

One of the seven principles of Lean Thinking is “eliminate waste.” Eliminating waste means minimizing the cost of the resources we use to deliver software to our stakeholders. Jean Tabaka proposes three pivotal practices that we must embrace to aggressively attack waste in software delivery —- Software as a Service (SaaS), Community, and Fast Feature Throughput:
Software as a Service (SaaS) eliminates waste by deploying software-based services without the cost inherent in traditional software delivery—materials, shipping, time delay, and more.

Community involves stakeholders working together to create products rather than competing among themselves for limited resources. Community eliminates waste by democratizing software development to obviate the need for multiple systems with the same functionality.

Fast Feature Throughput refers to development methods that embrace change and quickly deliver value to customers. It eliminates waste by responding to market pull with short, incremental delivery cycles.
When IT and all software organizations embrace these practices, they will eliminate waste within their organizations, reduce the waste that consumes our entire industry, and ultimately support the broad 21st century global mandate to manage our scarce resources.

I can't help but think how these same "pivotal practices" apply equally well to Agile CM, resulting (presumably?) in Software CM as a Service (SCMaaS), Community, and Rapid Change-Flow (where the latter refers to both quickness and responsiveness of change assessment and approval, as well as to development velocity as the changes flow through codelines and become integrated, built, promoted and released).

Rise of the Development Environment Architect

2008-04-15T01:33:00.005-05:00

Peter Eeles and I must be subconsciously on the same page. Because at the same time I was blogging about Software Architecture Views and Perspectives and Software Architecture Quality Attributes and their direct applicability to SCM/ALM solution architecture (and software process in general), Peter was working on an article for IBM developerworks entitled The Rise of the Development Environment Architect:

[Development] environments present challenges; and, interestingly, these challenges are similar to those of the systems they support. For example, development environments have to deliver against the required functionality and properties (such as performance and usability), often have to coexist with legacy systems (such as, in the case of a development environment, existing methods and tools), and have to acknowledge other constraints (such as the distributed nature of development teams, and existing skills and infrastructure).

All in all, creating a well-oiled development environment that accelerates, rather than hinders, project performance is a science unto itself. This is why IBM® Rational® has spent many years specifically developing a services capability that understands the challenges faced by organizations that 1) want to improve developer productivity, and 2) regard their development organization as a strategic differentiator, rather than simply a cost center.

Our experience has led the Rational team to define a role within the software development lifecycle called the "development environment architect." In October 2007, one hundred of Rational's most experienced development environment architects from across the globe gathered together in the first conference dedicated to this role to share their experiences. This article is a result of that conference and the discussions that took place.

As you read the concepts presented here, you may well question whether the development environment architect should be a role itself, or whether the individual or team who normally functions in the software or systems architect role should simply add consideration of the development environment to their list of architectural concerns. I believe that both propositions are valid. Furthermore, whenever the role of the "architect" is discussed, it is always qualified with the domain under consideration; thus we speak of a "building architect," "software architect," "systems architect," "enterprise architect," etc. The development environment is simply one of these domains, and one that is not traditionally a concern for the "software architect" role. I therefore believe that the "development environment architect" role is one that hasn't been emphasized before -- hence this article.

This article has several audiences and objectives. It is relevant to organizations undertaking an improvement to their development environment and who need to understand the value of a development environment architect to help guide their initiative. It is also relevant to those who are responsible for the technical content of the development environment -- i.e., development environment architects -- because this article introduces this responsibility as a role not previously defined. Finally, this article may supplement material contained within a development environment, in helping communicate its content, the role of its architect, and the benefits of having such a role in place.

Read the full article here. I may blog later about the similarities and differences between the sort of architecture that Eeles describes versus my 4+2 Views Model of SCM/ALM Solution Architecture

Update - July 2009: Peter gave a keynote presentation on this topic at RSDC2008 (PDF slides)

BOOK: Outside-in Software Development

2008-04-09T00:11:00.005-05:00

My review of Outside-In Software Development is in this month's edition of The Agile Journal.

Kessler and Sweitzer's Outside-in Software Development should resonate deeply with all those who genuinely value the principle of customer collaboration in the Agile Manifesto, and with anyone who has played the role of Product Manager for a software project. This 2008 Jolt award Finalist is not a book about eliciting or prioritizing requirements (or "user stories") for an Agile project. This book goes beyond mere user-stories and their ranking or velocity to focus on uncovering the underlying needs and goals of your stakeholders and understanding what truly adds value for the customer and the business.

... I think Outside-in Software Development is a profoundly important book for anyone in the Agile or Lean "camps" because it addresses and embraces the often neglected pieces of the customer-relationship puzzle that emerge from the stakeholders' perspective, often after the software is released. It shows us how many of those same Lean and Agile values of collaboration, responsiveness, waste-elimination, and respect for people can be successfully applied to the users' experience with our software, and to the stakeholders' experience with ourselves in the service of realizing the very business value we strive to deliver.

Read the full review.

BOOK: Programming Groovy and Groovy Recipes

2008-04-02T09:29:00.005-05:00

I just received an advance copy of Programming Groovy from the Pragmatic Programmer's Bookshelf. This complements their work that came out last month on Groovy Recipes.

From the Programming Groovy book webpage:

Groovy brings you the best of both worlds: a flexible, highly productive, agile, dynamic language that runs on the rich framework of the Java Platform. Groovy preserves the Java semantics and extends the JDK to give you true dynamic language capabilities⎯programming in Groovy feels like you’re using an augmented Java. Programming Groovy will help you learn and take advantage of the latest version of this rich dynamic language, so you can be a more productive Java Platform developer.

From the Groovy Recipes book webpage:

If you’re a busy Java professional who needs quick solutions to everyday problems, then Groovy Recipes is for you. The Groovy language and Grails web framework give you seamless integration with your legacy Java code while adding the flexibility and dynamism of a scripting language and giving you modern, agile, time-saving techniques. Groovy allows you to write code the way you always thought you should—you’ll never look at Java the same way again.

For those who like Ruby and Rails and the ability to access other Java frameworks and APIs, but also really want their Java-like syntax (and hence more than just JRuby), these are the books to read. Groovy even has its own answer to Rails called "Grails"

See also:

An Introduction to Groovy and Grails
Groovy with Grails – Java’s fight back to Ruby on Rails
Groovy Zone | Everything for the Groovy & Grails developer
AboutGroovy.com
Mastering Grails : Read more in this series to gain a further understanding of Grails and all you can do with it.
Groovy homepage
Grails homepage
Grails Framework Reference Documentation: The Grails bible.
Practically Groovy : This developerWorks series is dedicated to exploring the practical uses of Groovy and teaching you when and how to apply them successfully.
Grails Plugins: Documentation for all Grails plug-ins.

Brad Appleton's ACME Blog

SOA, Mashups, Mashed Knees and Surgery

BOOK: Running an Agile Project

WANTED: Seeking Single Agile Knowledge Development Tool-set

WANTED: Seeking Single Agile Knowledge Development Tool-set

Outline

Software development as knowledge development

How would I do this?

DISCUSSION & CHALLENGE: Why can’t (or how can) YOUR tools do this!

Studies on Effectiveness of TDD?

Resources on Retrospectives

Refactoring @ Scale

Refactoring for Agility

Mercurial, Git and Scala

BOOK: Landing the Tech Job You Love

BOOK: The Economics of Iterative Software Development

JEDI Programming - Just Enough Design Initially

Emergent Design and Evolutionary Architecture - Resources

Embracing Change - quotable quotes on change and uncertainty

Technical Debt - Definition and Resources

Resources on Self-Organizing Teams for Agility

Agile Self-Organizing Teams

Self-Organization and Complexity

Agile Self-Organization versus Lean Leadership

Value Proposition for Agility

BOOK: The Art of Lean Software Development

The Dynamics of Leadership-Team Behavior

Teams on the Way Down

Teams on the Way Up

BOOKS: The Passionate Programmer and the Nomadic Developer

5S Qualities of Well Designed, Well-Factored Code

HBR on Rebuilding Trust

Rewiring the Primal Management Talent Code

Synchronous and Staged Integration

The Five Traits of Agile Projects

Sensing and Sense-making in the Agility Cycle

BOOK: Agile Testing

BOOKS: The CMDB Imperative and the Data Access Handbook

An Agile CM Manifesto: Lame Oxymoron or Long Overdue?

Dee Hock on Hiring & Leadership

BOOK: Pragmatic Thinking and Learning

The Agility Cycle - Part 3

BOOK: Trust - The social virtues and the creation of prosperity

More Articles on Trust

BOOK: Building Trust

BOOK: The Speed of Trust

The Agility Cycle - Part 2

The Agility Cycle - Part 1

What is Agility? Part 2 - Software Agility

BOOK: Clean Code - A Handbook of Software Craftsmanship

CRISP Builds

What is Agility? Part 1 - Business Agility

What is Agility?

I'm ba-a-ack!

Gee Scrum Method Owners!

Summer of Books

Assigning Code Ownership-Policy Ownership

Four Rules for Simple Codelines

Traceability Matrix in an Agile Project

Iterative and Incremental redefined redux

The Laws of Codeline (Thermo)Dynamics

An Agile Approach to Release Management

BOOK: Software Teamwork - Taking Ownership for Success

Distributed Version-Control Guide on InfoQ.com

From PMBoK to Agility

BOOK: Implementing ITIL Configuration Management

Three Pivotal Practices to Eliminate Waste

Rise of the Development Environment Architect

BOOK: Outside-in Software Development

BOOK: Programming Groovy and Groovy Recipes