WANTED: Seeking Single Agile Knowledge Development Tool-set

I'll be presenting at Agile2009 in Chicago on the Tools for Agility stage on Tuesday 25 August, 4:45pm-5:30pm.


Here is my session description from the Agile2009 conference website (download PDF here)

WANTED: Seeking Single Agile Knowledge Development Tool-set

Aren’t code, backlog-items, tests, designs & documents all just different forms of system knowledge at different levels of detail? Why can’t the same tools help refactor, browse, search, and provide build/test automation for non-code forms of knowledge without requiring a separate tool/repository for each format? This talk is intended as a challenge to tool vendors/developers to see how this simple treatment of all non-code items as part of a single, unified project knowledge-base can be at once both immensely powerful, and imminently practical, without requiring too much added complexity.

Process/Mechanics

Approximately ~30 minutes of slides/presentation to “make the case” for why this approach is useful and desirable, followed by discussion of challenges (to and from tool developers/vendors, as well as the rest of the audience) as to its usefulness and benefits, and why their current tools can’t easily do so and why the should or should not be easy to enhancement them to do it.

Outline

Software development as knowledge development

• Source-Code as knowledge
• Requirements (Stories) and Tests as knowledge
• Other usable forms of project knowledge (e.g., build scripts & configuration, build/test result reports version control history/comments, online help & other supporting docs)

How would I do this?

• Refactoring Knowledge (thinking about the rest of the “knowledge” the way we think about the code, and its habitability, navigability, and structure/refactoring)
• Applying other agile-practices (like TDD, Continuous Integration, etc.) to non-code knowledge development
• Wiki-based skins, DSLs, and use-cases/design as domain-driven Wiki-word definitions
  • Patterns (giving things names), Retrospectives results and lessons learned
• Viewing, searching EVERYTHING (even the code) via wiki?
• The “Wu-Wei” of Traceability (Tracing without Tracing)
• Versioning and operating on wiki-like entities, just like with code (e.g., making “working copies”, branches and tags)

DISCUSSION & CHALLENGE: Why can’t (or how can) YOUR tools do this!

Begin audience discussion/dialogue: Why can’t a tool like Eclipse or a Wiki-based CMS (such as Confluence) be used as a front-end to browsing/refactoring and navigating ALL the knowledge of the system? (not just code, but stories, tests, build/config data, CI reports, backlog, retrospective lessons).
What makes it easy/hard for these and other tools like any of the following to do this, or to be simple “skins” or plug-ins on top of only a handful of tools instead of a whole kitchen sink full of separate tools and repositories.

• Eclipse, Confluence, Twiki?
• FIT, Fitnesse, Contour
• Doxygen / Javadoc
• Trac / Agile-Trac?
• Jira / Remedy

See also a blog-entry of mine entitled Can I have just one repository please?

Learning outcomes

• Thinking about Agile development as knowledge management & creation
• Current barriers to using agile techniques and supporting tools for non-code artifacts
• What do refactoring, TDD, continuous integration, etc. mean for non-code artifacts (and why you should care)
• Use of wikis for organizing, structuring and refactoring ALL system knowledge
• Why manual traceability is unnecessary (and even comes for free) when using such an approach

Primary target persona

• Tomas Tool-smith/Tool-vendor

Other target personas

• Billy, Business Analyst
• David, Agile Developer
• Patricia, Project Manager
• Tara, Tester

(download PDF here)

Best Kept Secrets of Code Reviews

The folks over at SmartBear software have written a nice little book entitled The Best Kept Secrets of Code Reviews. It's free if you go over to their webpage and ask for it (you have to fill out a registration form, and it takes a few weeks to arrive, but they havent spammed me at all since I registered with them a few months ago).

This is a pretty good book and it is VERY pragmatic! It is applicable to Agile development too! [You don't have to do Pair-Programming to be Agile! Pairing is part of XP, which is one particular agile method -- several other agile methods do not require it.]

SmartBear also has a pretty neat suite of tools that look to me like they would be REALLY USEFUL for an organization trying to streamline some of its otherwise heavyweight processes for peer-reviews and related quality metrics:

• CodeCollaborator - Automation for paperless peer code-reviews
  
• CodeReports - Continuous source code metrics over time.
  
• CodePickle - Suspend & resume code changes in local developer sandboxes (implements the PrivateVersions pattern without using version-control branches)
  
• CodeReviewer - automated peer-to-peer code reviews across remote sites
• CodeHistorian - Data-mining and visualizations for version control systems.

And "No!" they did not ask me to blog or say anything nice about them or their products! I'm simply coming from the perspective of someone in a large organization who has witnessed a lot of homegrown and heavyweight processes and tools for these kinds of things, and don't see too many commercial tools addressing the peer-review aspect of development and trying to make it lighter-weight and better-integrated with version-control and the rest of SCM.

The have some other nice resources too:

• Agile Peer Review: Methods, and Results (Powerpoint Presentation)
• The Case for Peer Code Review
• Brand New Information about Code Reviews
• 11 Best Practices for peer code review
• 2500 Reviews at Cisco® - The Largest Case Study of Code Review, Ever
• Why formal inspections fail
• Team-Building for the Cold, Dark, and Alone
• Compare 5 styles of review
• Pros & Cons of Four Types of Review
• Social effects of peer code review

Looks like a lot of "good stuff" to me!!!

Agile CM/ALM in 2007

Every January at CMCrossroads.com the CM Journal focuses on CM predictions for the coming year. My contribution for this year is Agile SCM January 2007 - Looking Back to Move Forward where I reflect on the previous two years' predictions and update them just a bit, with lots of emphasis on Rational Jazz and MS VSTS:

ALF and Corona? or Open-Sourced Jazz?
While there has been activity on the websites for the ALF and Corona projects, we havent been hearing so much about them in the media outside of Eclipse-centric conferences, and press-release announcements from yet another vendor with an Eclipse "compatible" release of their software. While this certainly doesn't shatter the prediction, there has been more attention (perhaps due to more marketing?) about the upcoming ALM collaboration framework from IBM Rational called "Jazz."

Based on the RDSC 2006 Day 2 Keynote presentation (and also at JavaOne), Jazz is touted as the next generation of the Eclipse platform and the team based software development platform. Rational will of course provide a tool-set to go with it, but says it plans to release the overall framework (and code/APIs) under Open-Source. The claim is that the result is an ALM collaboration architecture which is inherently scalable to provide true support for geographicall dispersed development projects in a true collaborative fashion. Well known names from Rational have been presenting Jazz at various conferences throughout 2006 and generating lots of "buzz," as evidenced by articles and blogs like the following:

• IBM is Revamping Rational
• IBM Jazzes Collaboration
• JavaOne 2006: Keynote on Agile Eclipse, Transparency, Honest, and Retrospective
• Rational Jazz Soon to Eat More of It's Own Dog-food: Transparency
• Jazz is the news
• IBM Jazzing up Rational Offerings, from InfoWorld
• IBM to employ open-source development style for tools, from eWeek
• Online Collaboration and Agile Software Development, from Dr. Dobbs Journal
• Big Jazz News on IBM developerWorks blogs
• Agile Requirements Engineering in the Jazz Collaborative Development Environment, from Rutgers 2006 Software Engineering day
• Hot Jazz vs. Cold Beans, blog-entry

Interestingly enough, with all that marketing buzz, Rational was also very clear about that fact that Jazz would not be released until after 2006. So look for even more "hype" and "hope" in 2007 when the first releases of Jazz become available, and especially when the core Jazz infrastructure is released as open-source under the auspices of Eclipse project. It will be interesting to see the impact this has on the Eclipse ALF and Corona projects (which Jazz doesn't currently use, althought it could conceivably make use of Corona in the future) and the Eclipse BIRT project (which Jazz does make use of). Current marketing would suggest that Jazz is the new vision, or more aptly, more likely, the new open-source hope to slay (or at least compete with) MS VSTS

The site founder, Patrick Egan, usually also initiates a discussion thread to engage the readership into the topic, and this year was no exception, as Patrick asked us What needs to change in CM and ALM for 2007. At the end Patrick added "How does CM fair in an increasingly Agile world?"

This sparked the perennial debate about "Agile CM" and whether or not it's any different from more traditional/mainstream CM. Folks wanting to read the whole thread are certainly welcome to do so. I'm going to excerpt what I think are some relevant thoughts from what I posted ...

My thoughts are much along the same lines as Frank's posting. I think we need to raise the level of abstraction of our thinking and problem solving. There are tools that do this, but the problem is that until the "lowest common denominator" tools catch-up, the majority of the state of the practice won't move too far past that (IMHO).

I do think its a good thing that two of lowest common denominator tools on the version-control side seem to be in the process of being supplanted with successors that go further in this direction (CVS -> Subversion, and VSS -> MS VSTS). I don't see as much on the change-tracking side, and integrations still aren't very integrated for the most part.

Regarding "an increasingly Agile world" ... as more and more shops and companies (and vendors) say they are "Agile", the more the term "Agile" becomes diluted and branded, and more of a marketing term than anything else. And that does do damage (as Joe mentioned); In fact it does the most damage to those that are doing it justice.

I do however think that the increased emphasis & visibility on frequent automated integration is a good thing, and that the trend of not only doing more of that, but using that as a mechanism of generating and reporting more (and more accurate) metrics is also a good thing. ...

The term "Agile" and the ideas behind it is no better or different from other "fads" like Object-Orientation or [software/design] Patterns. They had lots of hype and buzz, they had some legitimate improvements to contribute to the state of the practice (even if the ideas themselves weren't new - that's not necessary in order to have great impact/influence), and once everything was all "hyped out" they faded into the woodwork, as a natural, acknowledged, part of the fundamental knowledge we should use+apply everyday.

Increasingly, software CM implementations & deployments will need to better accommodate an increasingly Lean/Agile world (and it will be more important to know the difference between the pretenders and the real-thing). While I agree with Joe and others that the basic tenets of what CM is and what it is supposed to be don't change because of Agile, the fact remains that the norm for how CM is actually practiced and implemented in many shops will need to change.

Many tools/vendors are already headed in that direction, and have recognized the trend foretold by Thomas Friedman (The World is Flat), Peter Fingar (Extreme Competition) and others like Dan Pink, John Seely Brown, and more. With CollabNet, MS VSTS and Rational Jazz, the emphasis is on connecting and collaborating. And ways of doing software development and CM that fail to acknowledge that, and at least the first five of what Steve McConnell calls The Ten Most Important ideas in Software Engineering are going to have a tougher go of things than they have in the past.

As to whether or not "Agile CM" does (or will) exist, that will be determine by the people who end up actually doing it, and defining it. Anything claiming to be "Agile" will need to legitimately embrace the ideals of collaboration and the human element that Agile emphasizes, as well as at least the first 4 of the 10 ideas McConnell identifies. It will also need to embrace Lean principles and techniques. And if I get to have any part in shaping it, I think it should also include heaping helpings of Theory of Constraints (including Critical Chain) and Six-Sigma as they apply to CM and the "throughput" of change-flow, and change management.

All those things have something useful and important to offer us, and for those that can sift thru the hype and wannabees, there are huge gains to be had (whatever the trendy term end up being). I also agree with Bill Langston that there are some relevant standards (as well as "classic" ideas & methods) that are important to maintain, and which must "temper" the application of the newer and trendier (instead of always going back and forth between rigid and unyielding extremes).

Maybe it should be "Jeet Kune Do" CM instead of "Agile" CM :) Because we need to look at the new things and "keep what is useful" without abandoning the old so we can adapt and change as the speed of business and technology mandates.

Essential CVS and Global Outsourcing with VSTS

Just received two new books about version-control tools:

• Essential CVS, 2nd edition, by Jennifer Vesperman
  [See the sample online chapter containing the CVS quickstart guide]


• Global Outsourcing with Microsoft Visual Studio 2005 Team System, by Jamil Azher
  [See the online sample chapter on "The Business of Outsourcing"]

The Essential CVS, 2e book is one of the better CVS books available these days. I think I like it better than the classic one by Fogel, but not quite as much as the Pragmatic Programmers "Practical Version Control with CVS" (still - it's pretty close).

To be honest though, I really dont feel like CVS is very desirable among free Version-control tool offerings when we have the likes of Subversion, Monotone, Arch, and others that support the more recent paradigms and higher-levels of abstractions for working with project-wide streams (branches) and more.

The VSTS book is rather interesting. The "Global Outsourcing" parts of the title, and some of the corresponding content, would likely "turn off" a lot of folks. It even has a brief section about Agile development (to which, you'd think "global outsourcing would be anathema).

Mickey Gousset published a review of the book back in October, and it's worth a read. I mostly agree with the comments he makes. I think the book is pretty good, but there is another one coming soon that I expect I'll like a whole lot better, as well as several VSTS books available from Amazon.com.

Still, if you need to do a lot of distributed development across geographically dispersed sites, and want to use VSTS not just for its versioning capabilities, but also the tracking and coordination capabilities, this is probably the book to get.

Agile Tooling Survey Results

From Pete Behrens' Agile Executive Blog, the results to the Agile Tooling Survey they conducted in October are now available online at http://trailridgeconsulting.com/surveys.html:

With over 500 survey responses from 39 countries, we feel this survey
provides an excellent benchmark for where the agile movement is at
today and how we are using project management tooling to assist our
agile processes.

This report builds a corporate profile of companies that are following
agile processes today and then uses that profile to analyze how they
are using project management tooling to support various aspects of
their agile processes.

It's rather interesting to see what sorts of tools are being used for version-control, defect/issue/enhancement-tracking (DIET), and project planning & tracking, particularly when some high-profile Agilists would have us believe that (other than version control) Agile should "eschew" such tools.

I don't think the problem is the tools. I think the problem is most of them were/are made and used in a non-agile fashion that didn't have the agile way of working in mind. Now that there are some tools out there which do, it seems they are helpful after all :-)

Agile vs MDE: XP, AMDD, FDD and Color Modeling

The February 2006 issue of IEEE Computer is devoted to Model-Driven Engineering (MDE). MDE is actually a bit broader than MDA/MDD, because MDE (allegedly) covers more of the lifecycle, and corresponding process and analysis. Doug Schmidt's Guest Editor's Introduction to MDE is a pretty good overview of the current theory and practice and the obstacles to overcome.

A co-worker of mine is very interested in combining Agile methods with Model-Driven Engineering. He feels that the benefits of agility and of model-driven code-generation show tremendous promise as a breakhrough combination in productivity and quality and he is stymied that there aren't a lot more folks out there trying to do it.

He attended UML World in June 2005 and had some discussions with Scott Ambler (AgileModeling), Steve Mellor (co-creator of the Schlaer-Mellor OOAD method, and co-author of "Agile MDA", Executable UML and MDA Distilled), and Jon Kern, Agile MDA Evangelist (who helped Peter Coad launch TogetherSoft). He found most of what they had to say supported the possible synergy between Agility and MDA, but was very surprised to see AMDD folks and XP/Scrum folks throwing away their models once they had the code for it.

Upon hearing the above, I noted that Peter Coad is quite possibly the missing link between MDE and Agility:

• Coad created the agile method known as Feature Driven Development (FDD) as well as the field of "Color Modeling" with UML.


• Like MDE and MDA, FDD is very focused on domain-modeling, and using the model (rather than the code) as the primary form of "capture" for communicating knowledge about the system. [See my article "FDD is Football-Driven Development (and XP is Rugby)" for an explanation.]


• The TogetherSoft tool-suite was allegedly tailor-made to be an "Agile IDE" of sorts to facilitate doing FDD and Color Modeling.

The potential mismatch between MDA, with AMDD and XP-like Agile methods, is that:

• Full/pure MDA strives for 100% generation of all code and executables directly from the models.


• Ambler's AMDD, and "domain modeling" espoused by the likes of Robert Martin, Martin Fowler, and others in the XP community strives for "minimal, meaningful models", where they model only as needed, as a means of gaining more understanding, and then embed the knowledge gained into the code and/or tests.

I beleive FDD has the potential to bridge the gap. It strives for a comprehensive domain model, but from that point the code is written by hand (using coding practices that are traditionally non-Agile in nature, including strict-code-ownership, and formal code reviews/inspections). FDD doesn't say anything about using MDA/MDD techniques to auto-generate code, but the method is extremely amenable to doing exactly that.

Furthermore, doing so would remove a lot of the manual parts and practices of FDD that many consider to be the least "Agile". And much of the FDD "Color Modeling" patterns and techniques are very much the equivalent of refactoring and design-patterns that are currently used for code. See the end of this message for some more resources on Color Modeling.

In my own humble opinion, I think the "sweet spot" is somewhere in between 100% code generation and "hand-crafted" code. I realize that 100% is the ideal, but I'm thinking about the 80/20 rule here, and whether trying to eliminate that last 20% is perhaps not always practical.

I think the biggest barrier to doing that today is tools:

• The modeling tools are good at handling the structure, but not as much as the behavior.


• High-level programming languages like Java and C# and their IDEs are more convenient for specifying behavior (which is still largely textual in UML 2 and Action-Syntax Languages).


• It is extremely difficult to maintain the non-interface code for a "class" or "package" unless it is either 100% manually coded or else 100% auto-generated. If it is 50-50, or even 80-20, then the "nirvana" of seamless and reversible round-trip design to code and back just isn't quite there yet.

What would get us there and help close that gap? I think the "melding" of the IDE with the modeling tool is what is needed - and it would have to allow specifying code such as Java or C# as opposed to only allowing ASL "code" (most of which looks pretty darn close to Java and C# anyway :) as well as a means indicating if/how a chunk of code was auto-generated or if it was to be hand-crafted, but "navigable" and editable via the Model.

The Eclipse framework shows a lot of promise in helping us get to that point, and has a lot of the groundwork and building blocks already in place, but still has a lot more work to be done.

I hear some of you saying, "Okay Brad, I see what this has to do with Agility. But what does this have to do with CM?" Well, in my January 2005 Agile SCM column, among other "crystal-ball gazing" predictions, I talked a little about "Model-Driven CM" and how it would resurrect the once popular SCM research-area of Software/System Configuration Modeling:

• MDE/MDA would readily lend itself to allowing the developer to focus on the logical structure of the code, letting the physical structure (files and directories) be dictated by the code-generation tool with some configuration scripting+profiles.


• This in turn would allow models and modeling to be easily used to analyze and design the physical structure of the code, including build & configuration dependencies.

Of course, we have a ways to go until we get there, but I do believe the trend is on the rise and it's only a matter of time.

Some other resources related to Agility and MDE:

• Anyone going to EclipseCon2006 to see Agile Application Development with Eclipse and Pragmatic MDA
• Hans Wegener's Agility in Model-Driven Software Development? Implications for Organization, Process & Architecture
• Andreas Voelter's Patterns for Incremental/Iterative MDD
• Martin Fowler's Language Workbenches and MDA
• IEEE Software Oct 2003 Point/Counterpoint" on MDA -vs- MDD
• Jorn Bettin, on the Practicalities of Implementing CBD and MDA and where Agile fits in the picture
• Dion Hinchcliffe's Domain-Driven Design: MDD Done Right?
• Alan Cameron Wills' Agile Components - Scaling XP
• A 2004 Workshop on Graph rewriting for Model refactoring in MDE/MDA, sponsored in part by:
• Scientific Research Network on Foundations of Software Evolution
• A Formal foundation for Software Refactoring
• Architectural Resources for the Restructuring and Integration of Business Applications (ARRIBA)
• Research Center on Structural Software Improvement
• The Program-Transformation.org Wiki-Web and Homepage
• And lastly, Jeff Sutherland writes of Charles Simonyi's latest effort, Intentional Software:
  

  "Simonyi’s vision, and the reason he started his company, is to take the idea of models and go one step further: to link the model and the programming so tightly that they eventually become the same. Programmers and users will be able to switch between many contrasting views of the model they are creating, and revise programs at will simply by tweaking the models (see “Just-in-Time Programming,” below). It’s something like an architect being able to draw a blueprint that has the magical property of building the structure it depicts—and even, should the blueprint be amended, rebuilding the structure anew."

Book Review: Practical Development Environments

Matthew Doar's Practical Development Environments (PDE) looks to be a pretty AMAZING book. It really does cover the entire lifecycle of development environment tools for version control, build management, test tools, change/defect tracking, and more. My previous favorite work on this topic was the Pragmatic Programmer's Pragmatic Project Automation (PPA), but no more.

The PPA book is still a GREAT book! And it focuses a lot more on programming and automating tasks and good ways to go about doing it. It goes into some of the details of particular tools and setting them up, especially JUnit.

But the PDE book is far more comprehensive in the range of development environment practices and tools that it covers, including not just the automation aspects, but evaluating them, setup and administration, integrating them together (and issues and challenges encountered), and many more aspects of testing, building, project tracking, version controlling, and just generally helping the development team get work done with maximal support and minimal hindrance from the tools they use.

If you want to be a toolsmith, and learn more about scripting and automating tasks and some of the common tools that already exist, then I'd still recommend Mike Clark's Pragmatic Project Automation.

If you're focus is less on how/when/why to automate and more on evaluating, setting-up and maintaining a practical development environment for your team, then Matthew Doar's Practical Development Environments is definitely my top pick nowadays!

Polarion for Subversion

A quick follow-on to my previous blog-entry on Subversion plus Trac gives SubTraction and an even earlier one asking Can I have just one repository please? ...

I just heard of a new product called Polarion which allegedly appears to do almost exactly what I envisioned in my "just one repository" blog-entry, and there appears to be a "Polarion for Subversion" offering (which also claims to support Ant, Maven, and Eclipse):

"In classic software development tool environments, many different point solutions are used for software life-cycle management. There are requirements management tools, bug trackers, change management, version and configuration management tools, audit and metrics engines, etc. The problem: your development artifacts are scattered, making it difficult to derive useful, timely management information. POLARION® ... keeps all artifacts of the entire software life-cycle in one single place ... gives organizations both tools (for requirements, tasks, change requests, etc.) AND project transparency through real-time aggregated management information ... combines all tools and information along the Software lifecycle in one platform. No tool islands, no interface problems, no difficult, potentially fragile integrations anymore."

However, it does NOT appear to be opensource.

I'd LOVE to see a mixed commercial offering of, say, AccuRev, Jira and Confluence be able to provide this all in one package (just as I described in the blog-entry). [And with AccuRev's and Atlassian's roots in and commitment to opensource (the folks at AccuRev had previously developed the open-source CM system "ODE" for the OSF), they might even consider making it freely available for opensource projects (like Atlassian currently does for both Jira and Confluence)]

Hey! I can dream - can't I? :-)

Subversion plus Trac gives SubTraction

Here's a bit of a "plug" for some open source SCM tool offerrings ...

For those CVS users who don't already know about Subversion I urge you to take a look. Subversion was designed to be a next-generation replacement for CVS that has a lot of the same basic syntax and development model while fixing or updating most of its well known shortcomings.

Another spiffy open-source project that integrates with both CVS and Subversion is Trac, which provides simple but powerful defect/issue/enhancement tracking (DIET) using a Wiki-web interface, and readily integrates with both CVS and Subversion to add collaborative, low-friction request/activity tracking to your version control and can be used to track change-sets in the version control tool and associate them with change-tasks/requests in the tracking tool.

Using Trac with Subversion can help "subtract" a lot of the tedium of traceability from your day-to-day work and give more "traction" to your development efforts. So, in a way, Subversion plus Trac gives SubTraction :-)

Book Review: JUnit Recipes

I have a whole bunch of reviewer-copies of books that I've been intending to review for several months. So I'll be doing a number of book reviews throughout the remainder of this year, particularly titles from The Pragmatic Programmers and from Addison-Wesley Professional (who were nice enough to give me copies of the books).

Today however I'll be posting about a review fo a book from a different publisher. I did a review of the book JUnit Recipes for StickyMinds a few months ago. My summary of my review was:

JUnit Recipes should probably be mandatory reading for anyone using Java, J2EE and JUnit in the real-world. This comprehensive and imminently pragmatic guide not only conveys a great deal of highly practical wisdom but also clearly demonstrates and explains the code to accomplish and apply the techniques it describes.

The full review is featured this month on the StickyMinds front page and is available from their website at http://www.stickyminds.com/s.asp?F=S767_BOOK_4