Wednesday, October 04, 2006

Scaling Agility: Distributed Agile Development

Current issues of IEEE Software, CACM, and ACM Queue have articles related to agile distributed development and release management ...

The Sept/Oct 2006 issue of IEEE Software is about Global Software Development. It has several Agile-related articles (like A Practical Management and Engineering Approach to Offshore Collaboration)

This months CACM theme is "Flexible and Distributed Software Processes" with articles on distributed agile development (which are currently available online), including:
ACM Queue an article on Agile/Iterative Release Management entitled Breaking the Major Release Habit.

Other resources on Distributed Agile Development:

Also, although it's not specific to Agility, the book Software without Borders appears to have some good reviews by several folks who are well-respected in the Agile community (also check out the online references section of the book.
Posted by at No comments:
Labels: , , , ,

Saturday, September 30, 2006

Scaling Agility: Agile Program Management

Over the past months I've come across a bunch of good links & papers on the topic of "Going Agile" at the program-level:

Michele Sliger (of Rally Software Development) has several good articles and presentations on Relating PMBOK Practices to Agile Practices
On using Agile methods in organizations with a stage/gate approach to program management, see some of Per Runeson's work in this area:
Murray Cantor has some good papers on Governance and Variance as it applies to Agility:
Some other papers & resources:

Those interested in some advanced agile planning concepts should look at Jeff Sutherland's paper on Scrum II - The Future of Scrum: Parallel Pipelining of Sprints in Complex Projects (and the presentation slides that go with it)

There are several REALLY GOOD whitepapers on Adopting & Scaling Agile at Rally's Agile Knowledge Portal, including the following in particular:

There's gotta be some other good stuff out there and Agile Portfolio, Program and Multi-Project Management! If you know of any - please add a comment and hyperlink or URL!
Posted by at 6 comments:
Labels: , , , ,

Thursday, September 28, 2006

Scaling Agility: Agile Systems Engineering

Over the past months I've come across a bunch of good links & papers on the topic of "Going Agile" at the program-level for large systems and systems of systems. Some of these relate to Agile program Management and others are more about Agile Systems Engineering (and some relate to both). I'll mention the ones on Agile Systems Engineering in this blog-entry and leave the ones on agile program management for a subsequent entry:


That's the best I came up with. If you know of other good links on this topic, please send me a comment!
Posted by at 1 comment:
Labels: , , , ,

Tuesday, September 26, 2006

Scaling Agility: Adapting Agile to the Organization

Here is a list of resources I've found that I feel are applicable in figuring out how to scale Agility for a large organization and project. (On the subject of metrics and values, I personally find Sam Guckenheimers work to be of greatest interest):







Additions and corrections are welcome!
Posted by at No comments:
Labels: , , , ,

Thursday, September 21, 2006

Scaling Agility: Seamless Agility across the Enterprise

David Anderson writes about the recent Agile2006 conference in his blog-entry Thoughts for Agile2006:

Scaling Agile. The BIG issue for this year is scaling agile across a whole organization. I see this as having three parts - program or multi-project management and the rollup of schedules and resource plans to a Director or VP level; architecture and enterprise level modeling of a domain and data center; and finally configuration management including build, integration, branch and merge strategies, and work-in-progress batching and related communication.

Ive been dealing with this topic a LOT lately in my own organization as part of efforts to spread amd adapt Agile methods across a large distributed enterprise working with large systems and teams. Ive been researching and collecting lots of resources, including some earlier blog-entries on Agile CMMI and Dancing Elephants and Agile Adoption across the industry.

My perceptions of where the "seams" of the enterprise are that are hardest to introduce Agility into are the close collaboration and alignment required across organizational (lifecycle discipline) boundaries and geographic boundaries (and I find the former to be more difficult to surmount than the latter.)

If I try to categorize them as different areas or aspects that each require the ability to be agile, I come up with something like:
  • Process - Adapting Agile to the Organization (making processes responsive to change)

  • Product - Agile Systems Engineering/Architecture (making the requirements & architecture be responsive to change)

  • Project - Agile Program Management & Governance (making the project be responsive to change)

  • People - Distributed Agile Development (collaborating across multiple sites, teams, and timezones)

  • Organization - Agile Metrics/Reporting, Governance, and Organizational Design

  • Environment - Agile CM, deployment, operation/support, etc.

I'll be blogging separately with lists of resources of found for several of the above.
Posted by at No comments:
Labels: , , , , ,

Monday, September 18, 2006

TEA-Time - a metric to elicit TDD behavior?

I've been thinking about a metric that might elicit Test-Driven behaviors in my organization. As a first step to TDD, we definitely want folks to create automated tests as much as feasible and execute them frequently. Once they get that, I've been thinking about what sort of metric might encourage them to actually work in short, test-driven cycles, where requirements are elaborated test-by-test (given a use-case or story, write the first test, write the code to pass the test, refactor, rinse-lather-repeat).

Some of these folks are very much ingrained in a systems engineering V-model lifecycle that does a lot of big-requirements up-front. So ensuring they work to automate tests and execute them frequently isn't enough to enourage them to use an interative approach of fine-grained TDD-style elaboration. An idea I had for what to measure is something I chose to call Test-Execution Availability Time, or TEA-Time (I figure if nothing else, my friends in the U.K will like the name :-).

As proposed, Test-Execution Availability Time (or TEA-Time) would be defined as the mean time between when a system requirement description is first baselined, and the time at which a "sufficient" number of automated tests for the requirement were implemented and ready (available) to be executed.

I was thinking that if a group was measuring this and wanted to behave in a way that minimized TEA-Time, it might encourage them to elaborate requirements in an iterative and incremental fashion, in smaller and smaller "functional slices". One thing I'm not sure of is what "a sufficient number of automated tests" should be in the above.

Any thoughts or comments?
Posted by at 4 comments:
Labels: , , , ,

Friday, September 15, 2006

REVIEW: Practices of an Agile Developer

Are you a developer who wants to improve your personal development habits in a way that helps not just yourself, but also incrementally improves your project and your team? If so, then run, don't walk, and get your hands on Practices of an Agile Developer: Working in the Real World by Venkat Subramaniam & Andy Hunt, from the Pragmatic Programmer's Bookshelf.

Want to know more about why? Then read the complete review in the September issue of The Agile Journal (the theme for September is Collaboration and Reuse).
Posted by at No comments:
Labels: , ,

Tuesday, September 12, 2006

Agile CM on YahooGroups

As mentioned in an earlier blog-entry, I have created an Agile CM Yahoo Group. The description is:
The agile-cm group is for the discussion of ideas relating to Configuration Management for Agile development and of applying Agile concepts, methods and tools to the practice of Configuration Management itself (and CM "Patterns"). This includes aspects of CM that relate to agile development, refactoring & design patterns, agile project-management, Lean, Theory of Constraints (TOC), even SixSigma and CMMI to the extent that they can help CM and related practices to be more "agile".

As I wrote earlier, I hope this new agile-cm group will have a healthy balance of both SCM folks and Agile development folks so we can have some constructive multi-faceted discussions.
Posted by at 1 comment:
Labels: ,

Saturday, September 09, 2006

How I Blogged my Summer Vacation

Some of you may have wondered what the heck happened to my blog during August and the first half of September. Well - I lost some write-access to my website for a lot of the summer (while my host was recovering from a denial of service attack) and I couldn't login and update it.

I had several blog-entries "queued up" simply needing some final cleaning-up before I made them live, and that got interrupted, then I couldnt access them, then I had some vacations/travel, then I needed to "catch up" when I got back to work, and so on ...

So over the next week or two I'll be "pushing out" the entries I wrote-up in August, using the date they were originally written. I'll also be making available the talk I gave in July at Architecture & Design World 2006 on the subject of "SCM Patterns for Agile Architectures."

In October/November I may not be blogging much at all as I'll be trying to recover from major surgery (I'm giving one of my kidney's to a family member).
Posted by at 1 comment:

Monday, September 04, 2006

Relating SCM Patterns to SCM Principles

Our August Agile SCM column in the CM Journal is about Relating SCM Patterns to SCM Principles. The article is pretty flimsy, barely a skeleton, and that's pretty much entirely my my fault. I meant to write more "meat" about how certain principles are the underlying forces behind several patterns. I wanted to show how the principles are strongly related, much the same way patterns in a pattern language are related. And I wanted to show how that structure shaped the relationships between the patterns as well. Rob did a really good job trying to put together what I had with what he could come up with, but I didn't really give him enough and couldn't easily convey it in a way that made it easy for him to "run with it!"

It's not that I don't see those relationships, I do. And I'm not lacking for words to describe them either. I'm having trouble describing them clearly and concisely. A big part of that is because I still don't like the names of the SCM Principles as I've described them so far. Their names currently relate to the OOD Principles they were derived from. I think that might speak to programmers, but not to folks trying to do version control (even if they also wear "developer" hats). I really want to go back and rework the names of the principles to be more simple and direct.

This is something Ive been working in my mind on and off for alkmost a decade now. To me, it is of profound important - perhaps even the most significant contribution I'll have made to the field of SCM to date. THe interest level doesn't seem to be so high on the scm-patterns list and the cmcrossroads.com forums (but that hasn't deterred me - yet :). I really do think it's not a coincidence that principles of object-oriented design also manifest themselves as fundamental principles of SCM solution design (because I think both are all about architecture - and minimizing and managing dependencies).

I'd really like feedback. On this stuff so if you've had a chance to read thru the June, July and August Agile CM columns, I'm going to create a new YahooGroup about Agile-CM for discussing these and other Agile CM issues. Hopefully this new agile-cm group will have a healthy balance of both SCM folks and Agile development folks so we can have some constructive multi-faceted discussions.
Posted by at No comments:
Labels: , , , ,

Tuesday, August 29, 2006

More SCM Books in 2006

Some more SCM books have come on the market in the past 6 months or so. Several of them are worth a look: I realize Practical Perforce isn't new - I just can't stop gushing about how great it is! Laura Wingerd really has a gift for elegantly and simply stating important (and sometimes profound) insights about SCM, particularly when it comes to component/container-based SCM and codeline branching and management.

Global Outsourcing with VSTS looks interesting - not really sure if I like it enough to recommend it just yet. It certainly has an attention grabbing title.

The 2nd edition of Pragmatic Version Control using Subversion is just as outstanding as the first edition. It is my first choice for an introducitory book on Subversion and Version Control.

Kevin Lee's book on using ClearCase, ANT and CruiseControl is nothing short of outstanding. If you're a Java-shop using ClearCase and want lots of practical tips and tactics for doing automated builds and accelerating the build process, this book is a MUST HAVE! Kevin's website is also a great resource. Now that IBM/Rational has acquired BuildForge I imagine their will be updates at his website (and eventually a 2nd ed?) about using Clearcase with BuildForge (and maybe some stuff regarding Maven2?). He apparently has some new (online PDF-only) book out called The BuildMeister's Guide from a new series/publisher called Buildmeister books.

Pulsipher and Buckley did a great job in their earlier book The Art of ClearCase Deployment. Now they have a similar book about ClearQuest entititled Implementing IBM Rational ClearQuest: An End-to-End Deployment Guide. So far it looks very good. It's also the only ClearQuest book I know of - so If youre using ClearQuest for anything but a small or informal project you probably should get your hands on this book in a hurry!
Posted by at No comments:
Labels: , ,

Thursday, August 24, 2006

Everyone wants to be LUVEd!

I had a posting on Scott Berkun's project management (pmclinic) list that was rather well received. I thought folks here might be interested in it ...

Someone else had written:
> I've found that powerful people respect people who do not back down from them. Make your point forcefully.

I responded with:
That could backfire really badly! That might work with some, for others it will only make things worse. Either way, what they are looking for is respect for their perspective. They want to be LUVed:
  • Listened to,
  • Understood, and ...
  • Valued.
Making any assumptions about what it means to them to be LUVed and what behaviors they feel demonstrate LUV could easily be jumping to conclusions down a very wrong and unconstructive path.

Chances are the "annoyance" is because, somewhere, somehow, the two people involved have different notions of what it means to feel respected and to be treated respectfully and be LUVed. What you regard as disrespectful and disruptive may seem just the opposite to him. He may feel that by giving what you say substantial thought and vocalization on multiple sides to be an act of mindfulness rather than dissension.

I would suggest getting together one-on-one for an open, honest, and candid yet compassionate discussion about what each of you need and what you should/can expect from one another. Dont attribute feelings/motives/intent to him, instead describe his (or her) actual, factual, observed behavior and how it makes you feel, what that makes you think, and its consequences/impact on you and others.

Something is certainly out of alignment between the two people in question, but it may not be objectives or intent, but rather differences in values and belief as to what constitutes respect, progress (versus disruption) and what things are most/more important for overall success.

So in terms of advice, I would add an 'E' to the end of the LUV acronym to make it LUVE: Listen, Understand, Value/Validate, Empathize:
  • Listen attentively to what the other person has to say
  • Understand it to the best of your ability (you don't have to agree with it, just "seek first to understand")
  • Value/Validate what was said. That doesn't mean agreeing with it; It does mean finding the value in it and validating that value to the speaker.
  • Empathize with the speaker. Try to identify the feelings they are experiencing and demonstrate your understanding of it, and that it is okay for them to have those feelings.
I wish it was as easy to do habitually as it was to write about. If I could develop the above habit, I would be 10X more effective in both my personal and professional life!
Posted by at No comments:
Labels: ,

Saturday, August 19, 2006

Agile Adoption across the Industry

I've been needing to do some research into the apparent state of Agile adoption & usage across the industry. For those who are interested, here are the URLs I have gathered (some of them are pointers to excerpts or abstracts that you have to register and/or pay to read the entire article - but I found other URLs that summarized the information in them).

Agile Survey Results
Other Articles and Reports on Agile Adoption:
[updated 14 Sept with articles from AgileJournal.com and VersionOne.net]
Posted by at 2 comments:
Labels: , , ,

Monday, August 14, 2006

Relishing Ruby and Rails

I've been trying to keep-up with what's going on in the scripting world. Perl6 still isn't out yet. And Ruby and Groovy seem to be the hottest and most popular & promising.

Groovy is still trying to win support and evolve to a JSR implementation, and achieve a big following. Not clear yet if it's going to succeed or if it has stalled out. The attempted revival of the "Practically Groovy" series at IBM developerWorks suggests maybe it's still alive and kicking.

Ruby is positively on Fire! And Rails is a big part of the reason for that (though there is a similar "Grails" project for Groovy). I've been wading my way through books on Ruby. The best ones were from the PragmaticBookshelf, but now O'Reilly has a some good Ruby books too. So now my Ruby reading list is as follows (I'm about half-way through it):

So far, it seems like Ruby and Rails is an amazingly hyperproductive language and fraemwork that takes what many feel are the best of Smalltalk, Python, and Perl, makes it fun and elegant, and makes web development and enterprise integration for simple to glue together production-quality stuff in less than half the usual time.

So right now I would say that Ruby (and Rails) seems to be winning out over Groovy, and some (including Bruce Tate) even think it's beating out Java. I guess we'll see. Right now Ruby (with Rails) appears to lay claim to the title of scripting language with the fastest growing popularity and user-base.
Posted by at 1 comment:
Labels:

Saturday, August 12, 2006

Some Good SQL Books

I've been needing to "brush up" on my SQL and RDBMS skills (let's just say it's been a while :-). I found the following books from O'Reilly and Addison-Wesley to be extremely useful, in order of increasing usefulness:
The first two above are from Addison-Wesley. I found them to be good for a "quick review" for someone who used to do SQL and RDBMS extensively 10+ years ago but whose knowledge is a bit rusty and outdated.

The second two books above are from O'Reilly. I really like the two O'Reilly SQL books! I refer to them the most right now. The SQL Cookbook updated a lot of my existing knowledge and added to it, and the Art of SQL is fantastic! I just love the way it uses Sun Tzu's "Art of War" to apply to winning the "war" on DBMS and DBM complexity!
Posted by at No comments:
Labels:

Sunday, August 06, 2006

SCM as the Architecture of the Evolution of Architecture

One thing I particularly liked about my presentation of "SCM Patterns for Agile Architectures" at the Dr Dobbs Architecture & Design World 2006 conference in downtown Chicago...

I offered up several definitions or perspectives of SCM, including a traditional description, an Agile one, and an architectural one. I liked my architectural definition (I hadn't used that one before - I had used the other two in previous definitions). I said that, from an architectural perspective:

"Configuration Management is the Architecture of the Evolution of your Architecture!"


I rather liked that! I sort of used that to justify the title of the presentation, and presented an IEEE definition of architecture, along with Kruchten's 4+1 views model of software architecture. I then used that to justify my 4+2 model views of SCM/ALM solution architecture. I updated my drawing of the 4+2 model a bit. I think the addition of the Who+When+When+Where+Why+How shows how it may be a possible "bridge" between the Zachman Framework and the 4+1 views model.

Once again, the presentation is here. Feedback is welcome!
Posted by at No comments:
Labels: , , ,

Friday, August 04, 2006

SCM Patterns for Agile Architectures

Back in Mid-July I gave a presentation on "SCM Patterns for Agile Architectures" at the Dr Dobbs Architecture & Design World 2006 conference in downtown Chicago.

I gave a 90 minute presentation, and I had 100 powerpoint slides. I had no intention of covering all of the slides. I had the presentation split up into an intro, plus 4 subject areas. In "Agile" fashion, I planned to do "dynamic prioritization for fixed-scope time-boxing" by polling the audience to see which sections were most interesting to them (and which patterns within which sections were of greatest interest). I figured we'd get through the introduction (which has an updated picture of my 4+2 views model of SCM/ALM architecture), and then 2 of the 4 remaining sections (3 at the very most).

I ended up making it through all of the sections and finishing on time - but I couldn't have done it without skimming quickly over the individual patterns/slides that were of least interest for my audience. Overall, I think it went pretty well. I probably wasn't animated or charismatic enough in my presentation style and should vary my tone of voice a bit more. The pictures I used were very cool and very apropos, and very well received (which is good cuz I spent a TON of time finding ones I liked that had copyrights I could use).

For those who are interested, I've made the slides available here. I welcome your feedback in improving them!
Posted by at No comments:
Labels: , , , , ,

Sunday, July 30, 2006

The New Rules: Agile beats Big

The July 24 issue of Fortune Magazine has an article entitled The New Rules as the cover story, with the cover saying "Sorry Jack! Welch's Rules for Winning Don't Work Anymore (But We've Got 7 New Ones That Do)"

I think the new rules it discusses are very much about "Agile is better than Bigger!" and "Bigger isn't necessarily better!" The list of new rules is:
Old Rule: Big Dog Owns the Street.
New Rule: Being Agile is Best; Being Big can Bite You!

Old Rule: Be #1 or #2 in Your Market.
New Rule: Find a Niche, Create Something New.

Old Rule: Shareholders Rule.
New Rule: The Customer is King.

Old Rule: Be Lean and Mean.
New Rule: Look Out, Not In.

Old Rule: Rank your Players; Go with the A's.
New Rule: Hire Passionate People.

Old Rule: Hire a Charismatic CEO.
New Rule: Hire a Courageous CEO.

Old Rule: Admire my Might.
New Rule: Admire my Soul.
All in all I thought it was pretty fair-minded. There was even a sidebar to the article that gave Welch a chance to respond to the criticisms. You'll probably need to read the article for further insight into what exactly is meant by each of the "new rules" above. There was plenty of commentary across the industry on the article! (Just Google on "The New Rules" +Fortune +"Sorry Jack" and look through the results)
Posted by at No comments:
Labels: , , , ,

Monday, July 24, 2006

Agile SCM Principles - From OOD to TBD+CBV+POB

I finally finished a set of articles I'd been working on for almost 10 years on and off on the subject of "translating" principles of OOD into principles of SCM. See the following:
The principles of OOD translated into principles of Task-Based Development (TBD), Container-based Versioning (CBV), and Project-Oriented Branching (POB).

Here are the principles that I translated. Most of them are from Robert Martin's book Agile Software Development: Principles, Patterns, and Practices, but a couple of them are from The Pragmatic Programmers:


Here is what I ended-up translating them into. Note that some of the principles translated into more than one principle for version control because they applied to more than one of changes/workspaces, baselines, and codelines. I'm not real thrilled about the names & acronyms for several of them and am open to alternative names & acronyms:

    General Principles of Container-Based Versioning
    The Content Encapsulation Principle (CEP) All version-control knowledge should have a single authoritative, unambiguous representation within the system that is its "container. In all other contexts, the container should be referenced instead of duplicating or referencing its content.
    The Container-Based Dependency Principle (CBDP) Depend upon named containers, not upon their specific contents or context. More specifically, the contents of changes and workspaces should depend upon named configurations/codelines.
    The Identification Insulation Principle (IDIP) A unique name should not identify any parts of its context nor or of its related containers (parent, child or sibling) that are subject to evolutionary change.
    The Acyclic Dependencies Principle (ADP) The dependency graph of changes, configurations, and codelines should have no cycles.
    Principles of Task-Based Development
    The Single-Threaded Workspace Principle (STWP) A private workspace should be used for one and only one development change at a time.
    The Change Identification Principle (CHIP) A change should clearly correspond to one, and only one, development task.
    The Change Auditability Principle (CHAP) A change should be made auditably visible within its resulting configuration.
    The Change/Task Transaction Principle (CHTP) The granule of work is the transaction of change.
    Principles of Baseline Management
    The Baseline Integrity Principle (BLIP) A baseline's historical integrity must be preserved - it must always accurately correspond to what its content was at the time it was baselined.
    The Promotion Leveling Principle (PLP) Define fine-grained promotion-levels that are consumer/role-specific.
    The Integration/Promotion Principle (IPP) The scope of promotion is the unit of integration & baselining
    Principles of Codeline Management
    The Serial Commit Principle (SCP) A codeline, or workspace, should receive changes (commits/updates) to a component from only one source at a time.
    The Codeline Flow Principle (CLFP) A codeline's flow of value must be maintained - it should be open for evolution, but closed against disruption of the progress/collaboration of its users.
    The Codeline Integrity Principle (CLIP) Newly committed versions of a codeline should consistently be no less correct or complete than the previous version of the codeline.
    The Collaboration/Flow Integration Principle (CFLIP) The throughput of collaboration is the cumulative flow of integrated changes.
    The Incremental Integration Principle (IIP) Define frequent integration milestones that are client-valued.
    Principles of Branching & Merging
    The Codeline Nesting Principle (CLNP) Child codelines should merge and converge back to (and be shorter-lived than) their base/parent codeline.
    The Progressive-Synchronization Principle (PSP) Synchronizing change should flow in the direction of historical progress (from past to present, or from present to future): more conservative codelines should not sync-up with more progressive codelines; more progressive codelines should sync-up with more conservative codelines.
    The Codeline Branching Principle (CLBP) Create child branches for value-streams that cannot "go with the flow" of the parent.
    The Stable Promotion Principle (SPP) Changes and configurations should be promoted in the direction of increasing stability.
    The Stable History Principle (SHIP) A codeline should be as stable as it is "historical": The less evolved it is (and hence more mature/conservative), the more stable it must be.

You can read the 2nd article to see which version-control principles were derived from which OOD principles. Like I mentioned before, I'm not real thrilled about the names & acronyms for several of them and am open to alternative names & acronyms. So please share your feedback on that (or on any of the principles, and how they were "derived").
Posted by at 3 comments:
Labels: , , , , ,

Saturday, July 22, 2006

Agile SCM Principles - Design Smells

I finally finished a set of articles I'd been working on for almost 10 years on and off on the subject of "translating" principles of OOD into principles of SCM. See the following:
In an August 2005 blog-entry on SCM Design smells, I tried to translate the design smells that Robert Martin. wrote up in his book on Agile principles, Patterns and Practices. In the first of the two articles above, I think I was much more successful at translating them into the version-control domain. Here is what I came up with ...

Symptoms of Poor Version Control
Rigidity/Inertia:
The software is difficult to integrate and deploy/upgrade because every update impacts, or is impacted by, dependencies upon other parts of the development, integration, or deployment environment

Fragility/Inconsistency:
Builds are easily “broken” because integrating new changes impacts other fixes/features/enhancements that are seemingly unrelated to the change that was just made, or changes keep disappearing/reappearing or are difficult to identify/reproduce

Immobility/Inactivity:
New fixes/features/enhancements take too long to develop because configurations and their updates take a long time to integrate/propagate or build & test.

Viscosity/Friction:
The “friction” of software process against the development flow of client-valued changes is too high because the process has an inappropriate degree of ceremony or control

Needless Complexity/Tedium:
Procedures and tasks are overly onerous and/or error-prone because of too many procedural roles/steps/artifacts, too fine-grained “micro” tracking/status-accounting, overly strict enforcement or rigid and inflexible workflow

Needless Repetition/Redundancy:
The version-control process exhibits excessive branching/merging, workspaces or baselining in order to copy the contents of files, changes and versions to maintain multiple views of the codebase.

Opacity/Incomprehensibility:
It is difficult to understand and disentangle the branching/merging and baselining schemes into a simple and transparent flow of changes for multiple tasks and features developed from multiple collaborating teams working on multiple projects from multiple locations for multiple customers

In my next entry I'll start describing the actual principles and their translations.
Posted by at No comments:
Labels: , , , , ,

Thursday, July 20, 2006

Codeline Flow, Availability and Throughput

There has been an interesting discussion on codeline build+commit contention on the XP yahoogroup initiated by Jay Flowers' post about a proposed build contention equation ...

The basic problem is that there have been some commit contention issues where someone is ready to commit their changes, but someone else is already in the process of committing changes and is still building/testing the result to guarantee that they didnt break the codeline. So the issue isnt that they are trying to merge changes to the codeline at the same time, the issue is that there is overlap in the time-window it takes to merge+build+test (the overall integration process for "accepting" a change to the codeline).

Jay is being very agile to the extent that he wants to promote and sustain "flow" on the codeline (see my previous blog-entry on the 5 Cs of Agile SCM Codelines). He is looking at the [average] number of change-packages committed in a day, and taking into account build+test time, as well as some preparation and buffer time. Here the "buffer time" is to help reduce contention. It makes me think of the "buffer" in the Drum-Buffer-Rope strategy of critical-chain project management (CCPM) and theory-of-constraints (TOC).

Several interesting concepts were mentioned that seem to be closely related (and useful):
If we regard a codeline as a production system, its availability to the team is a critical resource. If the codeline is unavailable, it represents a "network outage" and critical block/bottleneck of the flow of value through the system. This relates to the above as follows:
  • Throughput of the codeline is the [average] number of change "transactions" per unit of time. In this case we'll use hours or days. So the number of change-tasks committed per day or per hour is the throughput (note that the "value" associated with each change is not part of the equation, just the rate at which changes flow thru the system).

  • Process Batch-size is all the changes made for a single change-task to "commit" and ...
  • Transfer Batch-size would be the number of change-tasks we allow to be queued-up (submitted) prior to merging+building+testing the result. In this case, Jay is targeting a one change-task per commit (which is basically attempting single-piece flow).

  • Processing-time is average duration of a development-task form the time it begins up until it is ready-to-commit. And ...
  • Transfer-time is the time it takes to transfer (merge) and then verify (build+test) the result.

  • Takt time in this case would regard the development as the "customers" and would be (if I understand it correctly) the [average] number of changes the team can complete during a given day/hour if they didnt have to wait-around for someone else's changes to be committed.

  • System outage would occur if the codeline/build is broken. It could also be unavailable for other reasons, like if corresponding network or hardware of version-control tool was "down", but for now let's just assume that outages are due to failure of the codeline to build and/or pass its tests (we can call these "breakages" rather than "outages" :-)

  • MTTR (Mean-time to repair) is the average time to fix codeline "breakage," and ...
  • MTBF (Mean-time before failure) is the average time between "breakages" of the codeline
Note that if full builds (rather than incremental builds) are used for verifying commits, then build-time is independent of the number of changes. Also note that it might be useful to capture the [average] number of people blocked by a "breakage," as well as the number of people recruited (and total effort expended) to fix it. That will helps us determine the severity (cost) of the breakage, and whether or not we're better off trying to have the whole team try to fix it, or just one person (ostensibly the person who broke it), or somewhere in between (maybe just the set of folks who are blocked).

Anyway, it's an interesting service-model of codeline availability and reliability for optimizing the throughput of codeline changes and maximizing collaborative "flow."

Has anyone ever captured these kinds of measures and calculations before? How did you decide the desired commit-frequency and how did you minimize build+test times? Did you resort to using incremental builds or testing?

I think that giving developers a repeatable way of doing a private development build in their workspace, even if its only incremental building+testing, gives developers a safe way to fail early+fast prior to commiting their changes, while sustaining flow.

I don't particularly care for the practice "build-breaker fixes build-breakage." At the very least I think everybody who is blocked should probably try to help (unless the number of people blocked is more than recommended size of a single team), and I'm sure the person who broke the build probably feels bad enough for causing the blockage (maybe even more so if multiple people help fix it). I think the build-breaker should certainly be a primary contributor to fixing the build and may be most familiar with the offending code, but they may need some help too, as they might not be as familiar with why/how the breakage happened in the first place since it slipped past them (unless of course it was "the stupid stuff" - which I suppose happens quite a bit :-)

So is anyone out their measuring serviceability, availability, reliability of their codelines? Are any of you using these other concepts to help balance the load on the codeline and maximize its throughput? I think that same of the more recent build automation tools (BuildForge, Maven, MS Build + TeamSystem, ParaBuild, etc.) on the market these days could help capture this kind of data fairly inobtrusively (except for maybe MTTR, and the number of people blocked and people+effort needed to effect the repair).
Posted by at 2 comments:
Labels: , , , , , ,

Saturday, July 15, 2006

The 5C's of Agile SCM

Lean has it's 5S technique. And while Im certain there's a way to translate those into SCM terms (which I may try and do someday, if someone hasn't already), I'm thinking about five important C-words for Agile SCM:
  • Correctness -- the property that the current configuration of the codeline executes correctly and passes its tests.

  • Consistency -- the property that the current configuration of the codeline builds correctly

  • Completeness -- the property that the curent configuration contains all that it should, builds all that it should, and tests all that it should

  • Cadence -- the property that the codeline has a regular rhythm/heartbeat/pulse that represents a healthy flow of progress (and creates a new resulting "current configuration" every time a new change is commited, integrated, and "promoted")

  • Collaboration -- the property that the balance of the above achieves a productive and sustainable degree of collaboration that serves as the source of value-generation
I think that the above represents all the properties that need to be present to a significant degree in order for the codeline to achieve smooth flow and accumulate increasing business value at a steady rate.

Am I missing anything? What about Concordance (via audits or with stakeholders)? Or Customer? Content? Context? (dare I use the word "Control"?)
Posted by at 2 comments:
Labels: , , , , , ,

Monday, July 10, 2006

Agile CMMI and Dancing Elephants

[updated June 1, 2007]

CMMI on the surface is definitely not very inviting to Agile. CMMI can be done in an agile fashion however. If CMMI is something you have a need for, then for secrets of how to do it "Agile-style", and details of success stories and lessons learned, take a look at the following links:

Also see "Integrating Agile Methods", and "Teaching the Elephant to Dance: Agility Meets Systems of Systems Engineering and Acquisition" (and others) from the CSE 2005 Annual Research Review.
Posted by at No comments:
Labels: , , ,

Friday, July 07, 2006

Trustworthy Transparency over Tiresome Traceability

If there was an Agile CM Manifesto, then this statement should be part of it!
Trustworthy Transparency over Tiresome Traceability

Note that my position on traceability is more "lean" than "agile" I suspect. I base this on the XP & Scrum centric views that were expressed in the March 2004 YahooGroup discussion thread Why Traceability? Can it be Agile? I think the "tests over traceability" is probably a valid summary of the XP/Scrum perspective from that thread.

I think myself and David Anderson would probably say something more along the lines of "transparency over traceability", where we acknowledge the important goals that traceability is trying to fulfill (I'm not sure the XP community embraces all of those that I identified as the "8 reasons" and "6 facets" that I identified in my paper on traceability dissected). David in particular has written in the past about "trustworthy transparency" and "naked projects" (projects that are so transparent and visible in their status/accounting that they seem "naked").

I also differ strongly with the many vocal opinions expressed in XP community when it comes to the use of tools for tracking requests/changes: I'm strongly in favor of using a "good" tracking tool. I think index cards are a great and valuable "tool" for eliciting dialogue and interaction with the "customer" (and I use them for this purpose, along with post-it notes). But I believe index cards simply do not "cut it" as a serious means of storing, tracking, sorting, searching, slicing & dicing development/change requests).

I do believe an extent of traceability is necessary, and that it's not necessarily "agile", but that it can be, and should be, "lean" and streamlined, and should serve the purpose of transparency, visibility and status-accounting rather than being a goal in itself. And I think there are several strategies and tactics that can be employed to achieve "lean" traceability in service to "trustworthy transparency and friction-free metrics."

I think that a "lean" approach of traceability would focus on the following:
  1. Flow: If one uses "single piece flow" and does changes at the granularity that TDD mandates, then software-level requirements, design, coding, and testing are all part of the same task, and tracking them to a single record-id in the change-tracking system and version-control tool would actually go a long ways toward traceability (its much more work & intermediate artifacts when those activities are all separated over time (different lifecycle phases), space (different artifacts) and people (different roles/organizations). When traceability efforts noticeably interfere with "flow" is when agilists will start screaming.

  2. Minimizing intermediate artifacts and other perceived forms of "waste" (overspecifying requirements or too much requirements "up front") because fewer artifacts means fewer things to trace.

  3. Collocating both people & artifacts (the former for communication, the latter for "locality of reference") for those artifacts that are deemed necessary.

  4. Coarse-Granularity and Modularity/Factoring of what is traced: tracing at the highest practical level of granularity (e.g., is it practical to trace to the individual requirement or the use-case? To the line of code, or to the method/subroutine, or to the class/module) - this would be about "simple design" and "(re)factoring)" as it applies to the structure of the traced entities and their relationships.

  5. Transparent, frictionless automation of the terribly taxing and tiresome tedium of traceability: focus on taking the tedium out of manual traceability and have it streamlined and automated as much as possible, ideally happening seamlessly behind the seems (like with Jane Huang's event-based traceability (EBT), or thru the use of a common environment "event" catcher within Eclipse or MS Team System server), probably using a task-based, test-driven (TDD), or feature-driven (FDD) approach.
Many of these concepts and more are embodied in Sam Guckenheimer's recent book on Software Engineering with Microsoft Visual Studio Team System. I found this book to be surprisingly good (outstanding even), and not at all what I was expecting given the apparent tool/vendor-specific nature suggested by the title. The value-up paradigm and most of the other concepts and values in the book are very well aligned with agility while still meeting the needs of more rigorous ceremony in their software and systems engineering efforts.

I'll close with a description of a recent presentation by David Anderson on Changing the Software Engineering Culture with Trustworthy Transparency:
"Modern software tooling innovation allows the tracking of work performed by engineers and transparent reporting of that work in various formats suitable for everything from day-to-day management and team organization to monthly and quarterly senior executive reporting. Modern work item tracking is coupled to version control systems and aware of analysis, design, coding and testing transitions. This makes it not only transparent but trustworthy. Not only can a tool tell you the health of a project based on the state of completion of every work item, but this information is reliable and trustworthy because it is tightly coupled to the system of software engineering and the artifacts produced by it.

The age of trustworthy transparency in software engineering is upon us. Trustworthy transparency changes the culture in an organization and enables change that unleashes significant gains in productivity and initial quality. However, transparency and managing based on objective study of reality strains existing software engineering culture as all the old rules, obfuscation, economies of truth, wishful thinking and subjective decision making must be cast aside. What can you expect, how will you cope and how can you harness the power of trustworthy transparency in your organization?"
As someone with a strong Unix and Open-Source heritage, I've long regarded Microsoft as "the evil empire" and loathed their operating system and browser and ALM tools. But in the last 3 years or so they've acquired a number of people in the Agile and ALM community that I highly respect (Brian White, Sam Guckenheimer, David Anderson, Ward Cunningham, James Newkirk) and the products these folks have worked on look incredibly impressive to me (even tho not all of them are still with Microsoft), plus I'm quite impressed with the whole of their Software Factories vision and approach ...

I actually may have to start liking them (or at least part of them :-). Don't get me wrong! I'm still a big fan Unix (and Mac OS/X), Open-Source, and more recently Eclipse, ALF and Corona; But the competing stuff from the folks in Redmond is looking exceedingly more and more impressive to me. Working on those kinds of things with those people would be an incredible experience I think (now if only I could do that without having to relocate from Chicago or spend 25% or more of my time traveling ;-).
Posted by at 1 comment:
Labels: , , , , , ,

Wednesday, July 05, 2006

Leadership/EQ Rites of Passage and the Mythical Manager Month

A bit of a follow-up on my previous blog-entry about Matthew Edwards and his recently published a book on Creating Globally Competitive Software: The Fundamentals for Regular People.

I wrote:
I have a lot of respect for Matt, he and I went thru a lot of "stuff" together over a very short+intense period (more on that in a moment) and managed to come through it while spreading a little bit of light. During that time I also pointed Matt in the direction of Agile development as a possible "way out of the madness", and he did his part to help make that a reality.
Here's the story on that ... I worked with Matt back in 1999-2002 on what was then a hideously dysfunctional "death march" project that we were trying to pull out of it's own self-created and self-perpetuated hole. The product was an internal one, and Matt, a former testing Guru, was one of my key customer reps. The project suffered from just about everything under the sun:
  • Bad management (failure to set+manage expectations & appropriate interfaces)
  • Dysfunctional customer & internal organization (warring tribes, turf wars, political silos, and a severe lack of trusting/trustworthy mgmt leadership),
  • Management that felt senior architects/designers aren't supposed to get their hands dirty in "coding"
  • A tech-lead with great technical & project knowledge/skill/experience and strong passion for quality design but with an equally great reluctance to lead, overly trusting and possessing piss-poor leadership & communication skills at that time (me)
  • Managers that had great communication skills, but no clue about successful software development, and no interest in learning it
  • A highly talented team of young, promising developers, but with a total lack of software development experience/maturity (which wouldnt necessarily be a bad thing if not combined with all of the above)
And so much more ... in fact that project managed to take two of the best-known worst practices ("the mythical man-month", and "too many chiefs/generals, not enough indians/soldiers") and combine them into an even worse one that I dubbed "The Mythical Manager-Month":
The Mythical Manager Month -- adding more managament to a late & failing project just makes everything thing worse and everyone more miserable.
I have to say, that project really taught me a lot about leadership and communication, particularly ...
  • how leadership differs from management, and from cheerleading
  • the importance of planning your communication and having a communication plan
  • the huge impact of really good managers versus really bad ones,
  • the difference between credibility and trust
  • the difference between power/influence and authority
  • how incredibly selfish, two-faced, and despicably unethical some folks can be
  • how to recognize malevolent manipulators who appear to "befriend" you to gain your trust, but will betray and backstab to get what they want
  • and how to recognize (and handle) a demagogue masquerading as a "heroic manager."
The first two years of that project were both a painfully magnficent failure, and a painfully magnificent teacher. It was definitely a leadership "rite of passage" for me, and leading the successful turnaround of project (in which agility played a large part) was a deeply educational and visceral personal experience that has largely shaped my career & objectives since.

The books by Patrick Lencioni on team dysfunctions and how to overcome them, as well as organizational silos, politics & turf-wars would have done me a world of good back then if they'd been available (and if I'd had enough previous appreciation of those problems to have read-up on them and other works related discovering and raising my Emotional Intelligence).

That project marked my transition from "unconscious incompetence" about leadership & communication to "conscious incompetence" and really motivated me to navigate the path to "conscious competence." I yearn for the day when it becomes unconscious competence.

I'm not quite there yet. It's been a long leadership journey (much longer in experience and learning than in actual years) since that project, and I still have a long ways to go. But these days my bookshelf at home is replete with just as many books about leadership, EQ, influence, and communication as my technical bookshelf at work is with books on software development, and I think about a lot more than just the technical strategies/techniques/practices and lessons learned in my day-to-day work.
Posted by at 3 comments:
Labels: ,

Monday, July 03, 2006

Creating Globally Competitive Software

A friend of mine, Matthew Edwards, recently published a book on Creating Globally Competitive Software: The Fundamentals for Regular People. I can't wait to get my copy and start reading through it.

I have a lot of respect for Matt, he and I went thru a lot of "stuff" together over a very short+intense period (more on that in a later blog-entry) and managed to come through it while spreading a little bit of light. During that time I also pointed Matt in the direction of Agile development as a possible "way out of the madness," and he did his part to help make that a reality.

Since then Matt has had a few other "gigs" that have advanced his experience and insights into software development (in a very Gerry Weinberg-esque fashion). He later co-founded Ajilus, which works and consults in global software development with a strong socio-technical perspective, having embraced the ideas of Agility, Scrum, Theory of Constraints, and systems thinking about the organizational/social roots of most seemingly technical problems.

So I'm really looking forward to reading what Matt has to say, as someone who has seen all of that from many perspectives, and has seen the light regarding agility, collaboration, organization, globalization and how to convey those lessons to "regular people." As part of his bio, Matt writes:
"I consult, teach, speak, write and deliver in the software solution delivery space with a focus on helping teams simplify the software delivery lifecycle - and deliver. Time, cost, team solidarity and structures, organizational behavior, ability to deliver, pulling projects out of the hole ... everything is interdependent and is usually social, not technical."
-- Matthew Edwards, http://www.ajilus.com/
Like I said, I'm definitely looking forward to reading through this one and seeing how it can help folks like me "connect" with "regular people."
Posted by at No comments:
Labels: ,
Subscribe to: Posts (Atom)