Showing posts with label Design/Arch. Show all posts
Showing posts with label Design/Arch. Show all posts

Thursday, August 20, 2009

SOA, Mashups, Mashed Knees and Surgery

Today is my birthday - I had arthroscopic knee surgery last night and am feeling pretty good so far (happy birthday to me). I know I still have a lot of meds/painkillers in my system and that its going to feel more uncomfortable the next few days. I'm still hobbling around on crutches but I'm feeling much more confident that I can attend (and present at) Agile2009.

I received some very good books on Mashups and SOA a few months back and I'm finally getting a chance to look at them in some more detail. They really are quite good! I think Mashups are the "latest frontier" for realizing the promise of SOA, and a natural evolution from Wiki-webs. Here are the books:

If you follow the links above to the InformIT.com homepage for each of the above books, you'll find some good excerpts and related articles that will give a preview of what to expect so you can judge for yourself.

Friday, July 17, 2009

Refactoring @ Scale

In my previous post, Refactoring for Agility, I posted an outline and some thoughts for Part I of an Overview on Refactoring. Now I'm ready to post on Part II which is about refactoring @ scale. By "at scale" I mean in the larger context of other agile practices, as well as for large projects.

PART II - REFACTORING @ SCALE

1. Scaling-Up
  • To scale refactoring for larger projects, some additional techniques & issues must be added to the mix.
  • Note that this is “in addition to” (not “instead of”)
    Refactoring In-the-SmallRefactoring @ Scale
    Small, Fast & Frequent RefactoringsLarger, Periodic & Planned Restructurings
    Emergent DesignIncremental Design & Evolutionary Architecture
    Deferred RefactoringRestructuring & Technical Debt
    Code SmellsArchitecture Smells
    Design Principles & PatternsSoftware Modifiability Tactics
    Simple/Clean CodeSupple/Domain-Driven Design

2. Emergent Design
  • Emergent Design is a fancy name for the resulting design that “emerges” from the synergy of combining Refactoring together with TDD, Continuous Integration and Automated Testing.
  • Emergent Design means that ...
3. Technical Debt [a.k.a. Design Debt]
4. Restructuring Technical Debt
  • If we accrue a non-trivial amount of technical debt, we can’t simply “refactor” it away.
  • Paying it off typically requires restructuring efforts (or even reengineering) that must be planned.
  • Iteration plans must accommodate specific tasks for these restructuring efforts (or even be dedicated to restructuring).
  • Ignoring it, or deferring it for very long is not a viable option!
5. Overview of Restructuring
  • Identifies higher-level issues (“architecture smells”) that typically represent violations of known principles of good software architecture & design.
  • Periodically applies larger-scale “refactorings” and/or many small refactorings that were previously deferred.
  • The goal is to “pay down technical debt” in order to limit the increasing costs of accumulated complexity.
  • Typically requires a concerted effort that must be separately planned.
  • Uses not only design patterns/principles, but also architectural patterns/principles, as well as software modifiability tactics.
6. Refactoring vs. Restructuring
7. Restructure Periodically
  • Restructuring is often associated with absent or neglectful refactoring and/or design.
  • But … Any large software project spanning multiple teams eventually needs restructuring.
    • Even in the presence expert-level architecture, design & continuous refactoring
    • This is just a reality of software evolution/entropy
  • Therefore … Large software projects should assume that periodic restructuring will be necessary, and should plan accordingly to:
    • Clean-up accumulated code-smells and apply numerous refactorings that were deferred but are now sorely needed,
    • Address architecture smells by applying restructurings, patterns and modifiability tactics that have broader impact.
8. Architecture Smells
See Stefan Roock's and Martin Lippert's book Refactoring in Large Software Projects (There was an earlier version of their work entitled Large Refactorings").
  • Smells in Dependency Graphs
  • Smells in Inheritance Hierarchies
  • Smells in Packages
  • Smells in Subsystems
  • Smells in Layers
9. Software Modifiability Tactics
  • Localize Changes (increase cohesion)
  • Prevent Ripple Effects (reduce coupling)
  • Defer Binding-time (defer decision-making)
10. When to Consider/Plan Restructuring?
11. Evolutionary Architecture
  • Software Architecture concerns infrastructure elements that must exist before you can begin execution.
    • Architecture is about things that are hard to change later, it is difficult to allow an architecture to emerge.
    • For large projects, this includes high-level organization of the system into functionality/elements that will be allocated to separate teams.
  • Key techniques of Evolutionary Architecture include:
    • Deferring Irreversible Decisions to the “Last Responsible Moment” (LRM Principle)
    • Architectural “Spike”, Architecture Iteration and/or Spike Iteration
12. Incremental Design
  • Design evolves incrementally, iteration by iteration, based on current business priorities and discovered technical limitations.
  • Incremental Design …
    • Does not prohibit thinking about higher-level design.
    • Does encourage planning in detail only what will be constructed soon.
  • Focus is on “Just Enough, Just-In-Time” :
    • Specifying too much detail too soon causes more rework later.
    • But doing less now and saving the rest for later should not require significantly more work later than it would today.
  • We must do Just Enough Design Initially to attain the right balance of anticipation and adaptation.
13. Just Enough Design Initially (JEDI)
  • Initial design (before coding) is still necessary.
    • This use of “JEDI” was coined by Stephen Palmer as part of Feature-Driven Development (FDD)
  • Basic rule of thumb to tell when “JEDI” is achieved:
    • At iteration-scope, when, after one pass through the iteration backlog, modeling in small groups does not produce any new classes or associations of real significance.
    • At task/TDD scope, when we have defined enough structure & interface(s) to know specifically what code to write/test, precisely where to write it, and exactly how to invoke it.
  • Techniques of “the JEDI way” include:
    • Collaborative Domain Modeling and Color Modeling [from FDD]
    • Supple Design techniques & patterns
    • Domain-Driven Design (a.k.a. DDD -- see domaindrivendesign.org)
    • Design Blitz & other Agile Modeling techniques (see agilemodeling.com)
14. Supple Design & DDD
  • Domain-Driven Design (DDD) approaches modeling the core logic of the software by focusing on the domain.
    • The basic idea is the design should directly reflect the core business domain and domain-logic of the problem to solve
    • This helps understanding the problem as well as the implementation and increases maintainability of the software.
  • DDD uses common principles and patterns as "building blocks" to model & create a "supple design”
    • Supple: pliant, malleable, limber, yielding or changing readily.
    • The design is firm yet flexible, with structure and intent both clearly conveyed and deeply realized by the code.
  • Patterns of Supple Design include: Intention-Revealing Interfaces, Ubiquitous Language, Side-Effect-Free Functions, Assertions, Conceptual Contours, Standalone Classes, Closure of Operations, Declarative Style

15. Resources:

Resources on Emergent Design and Evolutionary Architecture Resources on Restructuring Resources on Technical Debt Resources on Modifiability Tactics Resources on Supple Design & DDD

Thursday, July 16, 2009

Refactoring for Agility

Some of you might have guessed from my recent posts on Emergent Design, Technical Debt, JEDI Programming, and 5S Qualities of Well Designed, Well-Factored Code, that I've been looking into trying to teach the fundamentals of refactoring and how it scales to larger projects. I've gathered some references and quotes and some ideas for slides that I wanted to bounce around on my blog.

Here is an outline and some thoughts for part I of some slides ...

PART I - REFACTORING FOR AGILITY

1. Overview of Refactoring
  • Identifies design-maintenance issues (“code smells”) that typically represent violations of known principles of good design.
  • Incrementally and iteratively applies a set of design improvement techniques (“refactorings”).
  • The goal is to minimize complexity & duplication in order to maximize simplicity & ease-of-change.
  • Encourages the “right” design details to emerge “just-in-time” with minimal guesswork/rework.
  • Scaling-up includes the use of periodic restructuring, initial & incremental design (“just enough”), and evolutionary architecture.

2. Refactoring Defined [cite definition(s)]

3. Refactoring Myths -- Refactoring is NOT …
  • “Rework” – redesigning things that could, and should, have been designed correctly in the first place.
  • Design “gold-plating” – work that adds no business value, and merely serves to stroke the egos of perfectionists who are out of touch with business reality.
  • Miscellaneous code “tidying” – the kind that is “nice to have,” but should only happen when the team has some slack-time, and is a luxury we can do without, without any serious consequences.
  • A license to “hack” – avoiding any and all initial design & analysis and instead jumping straight to coding with no “real” design.
  • Reengineering – large-scale restructuring that requires a concerted effort over the course of several weeks/months to re-write or re-architect significant parts of the system.
4. Refactoring IS …
  • A systematic approach to source-code “hygiene” that minimizes the chances of introducing bugs
  • Improving the design of the code after it has been written
  • A behavior-preserving transformation of source-code structure
  • The process of simplifying & consolidating a work-product by making several, small, successive revisions focused on: preserving correctness, removing redundancy, revealing thoughts & intentions, and improving clarity & conciseness.
  • A disciplined way of making changes while exposing the project to significantly less risk.
  • An effective means to address the economic reality of software growth/complexity by reducing & amortizing its cost throughout the daily business of code development & maintenance activities.
5. Why Refactor?
6. How to Refactor
7. Rules of Clean Code
8. Rules for Simple Code
9. The Steps of Refactoring
10. Code Smells
11. Categories of Refactorings
  • Small Refactorings
  • Larger Refactorings/Restructurings
  • Each category contains as many as a dozen or more refactorings, most of which are catalogued at http://refactoring.com/catalog/

12. Refactorings (Some refactorings from real projects)
  • See http://refactoring.com/catalog/ for an up-to-date list (and the “Refactoring to Patterns” catalog too)
13. What to do if …?
  • I spot a “smell” that is not already known or catalogued?
  • There is no specific known/catalogued “refactoring” for what I think I need?
14. When to Refactor?
  • While adding functionality
  • While fixing a bug
  • While reviewing code
  • After coding the same/similar thing for the third time (to “factor out” the duplication)
  • A.k.a.: The Rule of Three: 3 strikes and you refactor.
  • After the third time you deferred refactoring a change, for any reason [The Rule of Three, again]
  • Before the end of the iteration if you haven’t been following The Rule of Three
15. Refactor Continually
16. When NOT to Refactor?
  • When the build is broken or tests don’t pass
  • When it would compromise meeting an impending deadline or commitment
  • When the code in question really just needs to be re-written “from scratch”
  • When it would modify code/interfaces that could significantly impact/break other work (e.g.: Published/public interfaces and protocols, Database schemas/tables/operations)
  • Sometimes we must defer refactoring for later and/or plan for subsequent restructuring
17. Refactoring to Patterns & Principles
Software Design Principles and Design Patterns are the underlying foundation for Refactoring:
  • Code smells (a.k.a “code pathologies”): Signal a possible violation of design principles, Suggest which refactoring may be needed
  • Refactoring: Correct a design principle violation (at least partially), Converge toward common design patterns
  • Design Patterns: Reconcile forces among conflicting design concerns,Restore balance between competing design principles
  • Design Principles: Lead us to attain desired design qualities/attributes
18. Design Attributes/Code Qualities
Qualities of Highly Maintainable Software:
  • Loose Coupling & High Cohesion
  • Hierarchy (Structural Decomposition)
  • Abstraction, Encapsulation & Modularity
  • Sufficiency, Parsimony and Primitiveness
  • Readability
  • Testability
  • Modifiability
  • Serviceability
19. Design Principles: SOLID, SoC, DRY, Shy
  • The SOLID Principles of Object-Oriented Design (from Uncle Bob)
  • The SoC Principle: Separation of Concerns — separate interface from implementation, policy from mechanism, behavior from construction, commands from queries, ...
  • The DRY Principle: Don’t Repeat Yourself (Eliminate Duplication), Single Point of Truth
  • The “Structure-Shy” Principle: (“Tell, Don’t Ask!”), The Law of Demeter, Principle of Least Assumed Knowledge
20. Other Acronyms of Simple/Agile Design
  • OAOO – Say Things Once And Only Once (restatement of the DRY principle)
  • DTSTTCPW – Do The Simplest Thing That Could Possibly Work! (restatement of the KISS principle)
  • YAGNI – You Aren’t Gonna Need It!
  • The LRM Principle: Defer Commitment of Irreversible Decisions to the Last Responsible Moment!
  • BDUF – Big Design Up-Front! (vs. JEDI)
  • JEDI – Just Enough Design Initially/In-front!
  • DDD – Domain-Driven Design
21. Design Patterns
22. Summary: Refactoring for Agility
  • Successively applies small behavior-preserving transformations to eliminate code smells
  • Based on proven design principles and patterns for achieving maintainability & modifiability
  • Good automated testing is a prerequisite
  • Refactoring is not rewriting, rework or restructuring
  • With refactoring, we continuously invest nominal effort to reduce the risk & cycle-time of changes
  • The goal is to minimize complexity & duplication in order to maximize simplicity & ease-of-change.
  • Practiced in a highly disciplined manner, it promotes:
    • Sufficient functionality
    • Simple & clean code
    • Supple design
    • Serviceable software
    • Sustainable team velocity

23. Resources:
Code Smells:
Design Principles:
Design Patterns:
- Online Resources:
- Books:
Other Agile Design Slogans:

Tuesday, July 14, 2009

Mercurial, Git and Scala

Three more brand new books I just received that are worth mentioning ...


For those who may not know ...
  • Mercurial (like Git) is an open-source, distributed version-control system.

  • While Scala is the "hot" new JVM-based programming language that not only contains the "best of both worlds" from other statically-typed languages (like Java) as well as dynamic languages like Groovy, but also the best of both worlds from object-oriented programming languages (OOPLs) and functional programming languages (like Erlang and Haskell). Plus it has message-passing and actor-based support for concurrency (like Erlang) while still being strongly typed and having access to all your favorite Java APIs (and anything else that compiles to JVM). Not too mention that Scala has a kick-butt Web development framework called Lift that is a next-generation framework to the likes of Rails/Grails and Apache Struts.

Scala is my first pick for the programming language that is most excitingly poised for the multi-core programming revolution in the age of multicore processors, cloud-computing, web 2.0 and access to all your favorite Java APIs/apps.

Monday, July 06, 2009

JEDI Programming - Just Enough Design Initially

I left a comment on the "What is Missing?" entry at the Agile-in-a-Flash blog. The author's asked the questioin "What is missing?" from the stack of Agile flashcards they are developing. I responded ...


I think the "JEDI" approach is missing (any by that, I don't mean the mantra of "use the source Luke" ;-)

I think there is something missing regarding TDD and Design. Uncle Bob's three rules of TDD (and other writings) often mislead people to think that there is ZERO design up-front, as is if NOT doing Big-Design-Up-Front (BDUF) implies that therefore there is zero up-front design (NoDUF).

This is false (and Uncle Bob has even vehemently said so in The Scatology of Agile Architecture) but how does a newcomer reconcile it with the three rules of TDD? I can't write test-code without being able to invoke the thing-under-test. I can't invoke a thing if I haven't attempted to design the interface.

If I design an interface (even for a single method/subroutine) I have to have some inkling of which class/package/module it would go in, at least INITIALLY! There is some initial amount of design I do before writing a test that is both necessary and sufficient to define "just enough" of the interface of what I want my test-case to test.

So I think that is what is missing, a card called "JEDI", for "Just Enough Design Initially."

To my knowledge, this particular definition of the JEDI acronym in agile development was first used by Stephen Palmer and other FDD luminaries at www.step-10.com and featuredrivendevelopment.com (just do a Google-search on "JEDI" AND "Just Enough Design").

I also think there is a relationship between JEDI and Eric Evan's " Domain-Driven Design (DDD), Supple Design (part of DDD), as well as *some* of the so-called "Pre-factoring". But it can be a risky, slippery-slope, so it would be great to have some guidance to help us know when we've done "Just Enough Design In-front/Initially."

I suppose JEDI is a way of straddling the "appropriate range" of risk between anticipation and adaptation. I envision some kind of graph or diagram with axes ...
  • On the left-hand side of overanticipating we have "too much too soon" and big/all up-front design.

  • All the way on the right-hand-side we have "too little too late." Here you are faced with legacy-rewrites, system re-engineering, large-scale restructuring, etc.
The problem with both extremes is the creation of technical debt:
  • One Extreme does it by adding complex structures and behaviors too early in the cone-of-uncertainty and causes too much rework to rewrite that which was not yet certain and subject to much variability.

  • The other extreme does it by sheer entropy/decay/rot that results from inattention and/or negligence
In the middle we have a delicate balance that we need to strike, and which JEDI represents: "Just enough" and "Just-in-Time." The problem is that this is a range, not a single perfect point. What makes the problem harder is that the range is different for different projects, depending on a number of factors (including scale and distribution). There is a certain amount of "pay-up-front" and "pay-later" that need to be balanced with "pay-as-you-go."

I imagine this "range" represents how to find the "sweet spot" that demarcates Just Enough Design Initially:
  • In the middle of the scale is pure refactoring. It is strictly emergent, pay as you go just-in-time by focusing ruthlessly on keeping code clean and simple.

  • Minor-to-moderate restructurings (too large to be refactorings, too small to be total re-writes) are to the right. Sometimes larger systems and/or systems constructed by multiple-teams can do a great job at refactoring and still not be able to avoid the need for minor-to-moderate restructuring, particular for aspects of the design that cut across teams and architecture and functionality.

  • Just to the left of refactoring would be so-called "pre-factoring", where you have enough experience refactoring that you are able to apply basic application of encapsulation, DRY, separation of concerns, etc. without adding premature abstraction of inessential complexity. This is hard to get right, and has risk associated with it. But it does get progressively less with the better judgment that comes experience and practice.

  • To the left of pre-factoring would be the subset of DDD known as Supple Design. And the rest of DDD would be to the left of supple-design.
Somewhere in this continuum might be "segments" corresponding to specific practices or techniques besides DDD and refactoring, such as clean code, simple/incremental design, and evolutionary architecture.

The trick of knowing the "just enough" range lies not just in experience and discipline, but also in understanding your context and your "lead-time". Lead-time in particular dictates how soon in advance you need to be able to think and anticipate (and at what level of detail). The shorter the lead/cycle-time, the less you need to anticipate and prognosticate.

So I would see JEDI as a card that somehow is able to depict this continuum between anticipation and adaptation and where these various techniques fall on it, and the factors/tradeoffs that help you identify the right "range" for yourself. (And of course there is a close relation to technical debt and the cost-of-change curve :-)

Anyone ever seen a diagram that bears any resemblance to what I'm thinking of here?

Thursday, July 02, 2009

Emergent Design and Evolutionary Architecture - Resources

As a bit of a follow-up to my earlier posting on Technical Debt - Definition and Resources I gathered some resources on the subject of Evolutionary Architecture and Emergent Design (which is closely related to refactoring, restructuring and reengineering).

Wednesday, June 24, 2009

Technical Debt - Definition and Resources

I ran across a few really good papers on the subject of technical debt that are fairly comprehensive in their treatment of not just what it is, but also how to manage it:
For those not already well-versed on the subject, Technical Debt [a.k.a. Design Debt] occurs when our software becomes difficult or risky to change, and takes increasingly more time & effort to evolve. Technical debt represents the cost of the accumulated amount of rework that will be necessary to correct and/or recover from the deviation between:
  • the current design of the system, versus ...
  • a design that is minimally complex yet sufficiently complete to ensure correctness & consistency for timely delivery.
This effort grows more than linearly over time as a system becomes bigger and more complex.


The economic impact of technical debt is directly related to the cost of complexity and its resulting “friction” against the velocity of the development team (and, subsequently, upon the ease of system evolution).

Technical debt can be caused by under-engineering just as much as it can be caused by overengineering (overdesigning). It is a difficult, delicate and dynamic balancing act to achieve the necessary and sufficient amount of design to implement only the essential complexity required by system:
  • Sometimes we knowingly (under great pressure) do something the "quick & dirty" way, with the intent to "do it right" later (but not too late).
  • Sometimes we try to do too much to soon, and elaborate or implement details of the requirements when we and our customers/users haven't yet learned enough about the true needs of the system.
  • Sometimes we haven't yet learned enough about good design, and unintentionally violate design principles, resulting in undesirable dependencies that make code or other work-products hard to change.
  • Sometimes we neglect to properly "tend" to the design and don't give it the necessary amount of ongoing care and feeding needed to keep it "fit" and "supple."
But whether it is a deviation between principles and practice (knowingly or unknowingly), guessing incorrectly, anticipating too early, neglect, poor quality, or even just the laws of software evolution, we must make plans and take action to deal with the inevitable entropy of evolution or else we will sink too far into technical debt.

That's just my two cents on the subject of course. What do others have to say about it?

Wikipedia defines Technical Debt by referring to the words of Ward Cunningham, who first drew the comparison between technical complexity and debt in a 1992 experience report:
"Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."

On the C2 Wiki, Ward defined Technical Debt as:
“Technical Debt includes those internal things that you choose not to do now, but which will impede future development if left undone. This includes deferred refactoring. Technical Debt doesn't include deferred functionality, except possibly in edge cases where delivered functionality is "good enough" for the customer, but doesn't satisfy some standard (e.g., a UI element that isn't fully compliant with some UI standard).”

Martin Fowler's definition of Technical Debt is also frequently cited:
"Doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future."

In his introduction to Refactoring to Patterns, Joshua Kerievsky quite simply defines design debt as follows:
Design debt occurs when you don't consistently do three things.
  1. Remove duplication.
  2. Simplify your code.
  3. Clarify you code's intent.
Few systems remain completely free of design debt. Wired as we are, humans just don't write perfect code the first time around. We naturally accumulate design debt ....

Due to ignorance or a commitment to "not fix what ain't broken," many programmers and teams spend little time paying down design debt.... In financial terms, if you don't pay off a debt, you incur late fees. If you don't pay your late fees, you incur higher late fees. The more you don't pay, the worse your fees and payments become. Compound interest kicks in, and as time goes on, getting out of debt becomes an impossible dream. So it is with design debt.

James Shore describes Design Debt in much the same manner:
When the cost of change increases, it's because the design quality is decreasing. Since retiring or rewriting software means writing off a huge investment, you wouldn't expect any team to let design quality deteriorate to that point. Yet it happens all the time. Why?

Design debt" explains the problem. When a team is working under pressure, they take shortcuts that compromise design quality. It's like taking out a high-interest loan. The team gets a short-term boost in speed, but from that point forward, changes are more expensive: they're paying interest on the loan. The only way to stop paying interest is to pay back the loan's principle and fix the design shortcuts.

Steve McConnell defines Technical Debt as follows:
“Technical Debt” refers to delayed technical work that is incurred when technical short cuts are taken, usually in pursuit of calendar-driven software schedules. Just like financial debt, some technical debts can serve valuable business purposes. Other technical debts are simply counterproductive. The ability to take on debt safely, track their debt, manage their debt, and pay down their debt varies among different organizations. Explicit decision making before taking on debt and more explicit tracking of debt are advised.

Mary Poppendieck gives a definition of Technical Debt in her upcoming book Leading Lean Software Development:
“All successful software gets changed. So if we think we’re working on code that will be successful … we need to keep it easy to change. Anything that makes code difficult to change is technical debt. Just like any other debt, the cost of paying off technical debt gets more and more expensive over time. … Technical debt drives the total cost of software ownership relentlessly higher … eventually we will have to pay it off or the system will go bankrupt.”

Kane Mar, in Technical Debt and Design Death, describes technical debt and later likens it to the effects of entropy:
Technical debt is simply defined as deferred work that is not directly related to new functionality, but necessary for the overall quality of the system. Examples of this include delaying upgrades to necessary tools and frameworks, delaying the refactoring of overly complex system components, and so on. If this technical work is necessary for the health of the system, then what happens when it is ignored for too long? What happens when several layers of cruft accumulate and there are no unit tests to aid refactoring? ...

In thermodynamics, “entropy” refers to the randomness of the components of a system. When the term is applied to software it is considered a measure of disorder. Thus entropy in software is the result of changes made to the code base, including bug fixes, updates to existing functionality, and the addition of new functionality. But over a period of time, these small changes can snowball to create a system that is difficult to change, overly connected to external systems, and lacks clear delineation of functionality.

Chris Sterling, in his upcoming book on Architecture in an Agile Organization, distinguishes between Software Debt and Technical Debt:
Software debt accumulates when focus remains on immediate completion while neglecting changeability of the system. The accumulation of debt does not impact software delivery immediately. At first it creates a sense of increased feature delivery with management, business stakeholders and the team. Business stakeholders respond well to the pace of delivered functionality. What they don’t understand is that this only represents an illusion of earlier returns on their investment.
...
This allows both the business and software delivery teams to live in the illusion of status quo far longer than they should. At some point previously small forms of decay in the system become large enough to affect our software delivery to the point that working harder and longer doesn’t result in success.
...
Debt is made glaringly visible when the team works on stabilizing the software functionality late in the release cycle. Integration, testing, and bug fixing is unpredictable and does not get resolved adequately before the release. People involved in the project stay late working to get the release out the door. It is now too late to pay back the debt accrued during the feature development.

The following sources constitute what I call software debt:
  • Technical Debt: those activities that a team or team members chooses not to do now and will impede future development if left undone
  • Quality Debt: diminishing ability to verify functional and technical quality of entire system
  • Configuration Management Debt: integration and release management become more risky, complex, and error-prone
  • Design Debt: cost of adding average sized features is increasing to more than writing from scratch
  • Platform Experience Debt: availability and cost of people to work on system features are becoming limited

Over at the Agile-in-a-Flash blog, Jeff Langr and Tim Ottinger provide a flashcard of tips & truths about technical debt, along with a more detailed discussion about its nature and origins. They distill several nuggets of wisdom like:
  • Incurring technical debt means your velocity slows down and you will deliver less value
  • The cost of getting out-of-debt is compounded over time: the longer you wait, the faster it grow!
  • If you plan to incur technical debt, the persons responsible must have a workable plan to pay it off!
  • “Interest only” payments won’t improve things
  • Pay early, pay often, and pay-as-you-go. (The only other options are bankruptcy or death.)
  • Remember: Those with the worst debt problems often have the most difficulty imagining a life without borrowing!

So how might we properly measure/monitor and account for technical debt? Well, some of the above papers I mentioned have some good ideas. Some that address it more explicitly are:
Here are a number of other very good resources on technical debt culled from blogs, articles, and presentations over the years. Some of them describe it, while others delve into methods for managing it and paying it off:

Sunday, June 21, 2009

Gee Scrum Method Owners!

[NOTE: I originally wrote this before Agile2009 (in Chicago), but forgot to press "publish", so it stayed a 'Draft' until I discovered this and published it much later]

Sung to the tune of Gee, Officer Krupke! from West-Side Story -- Music: Leonard Bernstein, Lyrics: Stephen Sondheim (see video of the movie performance)

Background:
For about a year now, I believe there has been a greater than usual amount of Agile-related blogging and mail-list discussion on the subject of the shortcomings of what is currently the most popular agile development method: Scrum. Some of them are legitimately about Scrum, others are really more about the kinds of problems one encounters introducing Agile methods without appropriate emphasis/execution of the technical practices (especially Refactoring and TDD).

It seems much of it may have started around the same time that Alan Shalloway (and later others, including Ron Jeffries, below) were banned from the
ScrumDevelopment list for “going meta” and discussing problems with and/or additions to Scrum itself instead of how to apply/use it. Alan subsequently created the Lean-Agile list for such discussions (and is also working on a book).
As of this writing, more recent examples include the following:

Cast of Characters: You will need individual actors portraying the following roles!
NOTE that the people are real, but the dialogue is fictional, even tho it is derived from bits and pieces of actual messages and blog-posts.

Lyrics:
    RON (spoken): Hey, you!

    ALAN (spoken): Me, Officer Jeffries?

    RON (spoken):
    That’s Scrum Master Jeffries to you! Gimme one good reason for not draggin’ ya to the scrumdevelopment list to answer for blamin all yer troubles on Scrum. What’ll you say to Schwaber, ya punk!

    ALAN (sings):
    Dear kindly Scrum-Lord Schwaber, You gotta understand,
    It's just our agile nature that gets us Scrum-list banned.
    Our management is clueless, our backlog’s full of junk,
    Golly Moses, that’s where Scrum has flunked!

    ALAN with BRAD & PETER:
    Gee Scrum-method owners, we're very upset;
    Your method doesn’t scale and we’re in technical debt!
    We ain't no extremists, Our motives are clean,
    Deep down inside us there is Lean!

    ALAN: There is Lean!

    ALL:
    There is Lean, there is Lean, There is untapped Lean,
    Deep inside XP and Scrum there’s Lean.

    RON (spoken -- imitating Scrum list-owner):
    That's an off-topic discussion!

    ALAN (spoken):
    Lemme tell it to the world!

    RON (“Scrum list-owner”):
    Just tell it some place else!

    ALAN (singing, while creating the Lean-Agile list):
    Dear Lean-Agile list-readers, The Scrum-list is too closed.
    Despite their Agile leaders, the truth must be exposed:
    They didn't want discussion of what Scrum oughta add.
    Leapin' lizards, that's why I'm so “rad!”

    ALAN:
    Hey Scrum-method owners, you're really deep-fried!
    There’s plenty wrong with Scrum that needs Lean-thinking applied!

    PETER
    : Those darn product-owners have gotta be curbed!

    RON: They’re pathologic'ly disturbed!
    BRAD: They’re disturbed?

    ALL:
    They're disturbed. They're disturbed, They're the most disturbed,
    Like they're pathologic'ly disturbed.

    ALAN (spoken, imitating Mary Poppendieck):
    Hear ye! Hear ye! In the opinion of Lean-thinking, these POs act depraved on account of they ain't had a proper “frame.”

    BRAD (spoken): Hey, they’re depraved on account of their dis-functional context!
    RON (spoken): Lemme to introduce ‘em to my fully functional foot!

    BRAD (singing, to Ron)
    My PO is so dastard, da coach won’t hear my plea,
    da team’s code ain’t refactored, ‘cuz PO won’t agree.
    They add it to the backlog for later to address,
    Goodness gracious, dat's why it’s a mess!

    RON:
    Please! “Bad product-owners!” that reason’s so lame!
    You shouldn’t even ask them, (and you should be ashamed)
    It’s really yer context that oughta be changed,
    It’s economic’ly deranged!

    BRAD: We’re deranged?

    ALL:
    We’re deranged! We’re deranged! We are so de-ranged,
    We're so economic’ly deranged!

    RON (spoken):
    In my opinion, Scrum don't need to have its head roles re-thunk at all. Use of RASCI matrices is surely a management disease!

    BRAD (spoken): Hey, I got a management disease!

    ALL (spoken sarcastically): So go buy an agile management tool.

    BRAD (singing to "Agile tool-vendor"):
    Dear kindly Scrum-tool master, they say go use your tools.
    They’ll save us from disaster, ‘cuz they’ll enforce da rules.
    And for our retrospectives, they’ll generate nice charts
    Glory Osky, that's why our code smarts!

    UNCLE BOB (with RON):
    Yikes! Stupid tool vendors, you've done it again.
    This team don't need a tool, just index cards and a pen.
    It ain't just enough to inspect-and-adapt;

    UNCLE BOB
    : Value craftsmanship over crap!

    BRAD: Ship 'er crap?

    ALL:
    Don’t ship crap! Don’t make crap! Craftsmanship over crap!
    Craft clean code as if you give a crap.

    UNCLE BOB (spoken):
    We hereby declare this project’s code to be crap on account of its ignerantive and excremental development. Failure to write clean code is professionally irresponsible! TDD and refactoring are not optional!

    RON: The problem is your context!
    ALAN: The answer is go Lean!
    BRAD: The problem is design debt!
    BOB: The answer is code clean!
    PETER: The problem is Scrum's growing!
    ALAN: The answer is Scrum's grown!

    ALL:
    Scrum Lords we got troubles of our own!
    To all unclean-coders We just have to yell,
    “Your projects can’t be agile if your code-bases smell!”
    Gee, Scrum method-owners, What are we to do!
    Dear Scrum method-owners, SCRUM YOU!!

Acknowledgements:


I credit Ron Jeffries with my inspiration (obsession?) in crafting the above! Ron quoted a line from the West-Side Story song “Gee Officer Krupke!” in one of his email replies and I couldn’t get the song out of my head all that day and began piecing together the above (which I developed iteratively & incrementally using TDD and refactoring – of course :-).

Tuesday, April 15, 2008

Rise of the Development Environment Architect

Peter Eeles and I must be subconsciously on the same page. Because at the same time I was blogging about Software Architecture Views and Perspectives and Software Architecture Quality Attributes and their direct applicability to SCM/ALM solution architecture (and software process in general), Peter was working on an article for IBM developerworks entitled The Rise of the Development Environment Architect:

[Development] environments present challenges; and, interestingly, these challenges are similar to those of the systems they support. For example, development environments have to deliver against the required functionality and properties (such as performance and usability), often have to coexist with legacy systems (such as, in the case of a development environment, existing methods and tools), and have to acknowledge other constraints (such as the distributed nature of development teams, and existing skills and infrastructure).

All in all, creating a well-oiled development environment that accelerates, rather than hinders, project performance is a science unto itself. This is why IBM® Rational® has spent many years specifically developing a services capability that understands the challenges faced by organizations that 1) want to improve developer productivity, and 2) regard their development organization as a strategic differentiator, rather than simply a cost center.

Our experience has led the Rational team to define a role within the software development lifecycle called the "development environment architect." In October 2007, one hundred of Rational's most experienced development environment architects from across the globe gathered together in the first conference dedicated to this role to share their experiences. This article is a result of that conference and the discussions that took place.

As you read the concepts presented here, you may well question whether the development environment architect should be a role itself, or whether the individual or team who normally functions in the software or systems architect role should simply add consideration of the development environment to their list of architectural concerns. I believe that both propositions are valid. Furthermore, whenever the role of the "architect" is discussed, it is always qualified with the domain under consideration; thus we speak of a "building architect," "software architect," "systems architect," "enterprise architect," etc. The development environment is simply one of these domains, and one that is not traditionally a concern for the "software architect" role. I therefore believe that the "development environment architect" role is one that hasn't been emphasized before -- hence this article.

This article has several audiences and objectives. It is relevant to organizations undertaking an improvement to their development environment and who need to understand the value of a development environment architect to help guide their initiative. It is also relevant to those who are responsible for the technical content of the development environment -- i.e., development environment architects -- because this article introduces this responsibility as a role not previously defined. Finally, this article may supplement material contained within a development environment, in helping communicate its content, the role of its architect, and the benefits of having such a role in place.


Read the full article here. I may blog later about the similarities and differences between the sort of architecture that Eeles describes versus my 4+2 Views Model of SCM/ALM Solution Architecture

Update - July 2009: Peter gave a keynote presentation on this topic at RSDC2008 (PDF slides)

Monday, March 31, 2008

Software Process-Line Architecture and Common Processes

Extending the analogy of software architecture views and quality attributes for software process architecture, I'd like to spend some time discussing how software product lines relate to software process architecture and "common processes" across an enterprise.

Many organizations strive for standard common processes, often as part of a CMM/CMMI-based process improvement. All too often I have seen the mantra of "common process" misused and abused to make the practitioners serve the process instead of the other way around.
Processes don't create great software. People and Teams do!
And while the process needs to meet the needs of the business and the needs of the customer, it has to first and foremost serve the needs of the practitioners so that they in turn may better serve the needs of the business to deliver operational business value.

Many in management seem to have the mis-impression that "common process" means "no tailoring" and everyone does everything the same way across products and projects throughout the organization. Process variation across products and projects is regarded as something to eschewed and stamped out, beating the offenders into compliance with top-down dictates and mandates and sanctions. If everyone does everything the same way then the people are more or less "plug-and-play replaceable" and can quickly and easily be reallocated to another project or product with zero learning-curve and associated start-up costs.

This is a dangerous myth that causes irreparable harm to process improvement and common/standard process efforts. Anything that focuses on individuals and interactions as subservient to common processes and standard tools is doomed to fail, and those organizations often end-up with the processes they deserve (along with many disgruntled, frustrated workers).

The purpose of such common processes and tools is not to be a rigid restrictive straightjacket for replaceable people. The intended purpose is to recognize that such people are irreplaceable and to provide a flexible knowledge framework to guide and enable them as the help each other collaborate to learn, grow, and lead in the discovery, practical application and effective execution of practices and improvements that are the best fit for a particular product, product, community and business-environment.

The intended purpose common software processes is quite simply that of process and knowledge reuse! And as such it shares many of the same fundamental problems and solutions as that of software reuse. Indeed it could even be argued that software process reuse is but a special case of software reuse. And current prevailing industry wisdom on the subject suggests that software product-lines show the greatest promise of leveraging software reuse for greatest business value.

In software reuse, we seem to recognize that "one size does not fit all." We acknowledge that even though different products, components, and platforms may share common features, that each one may have different project parameters and environments with different quality attributes and engineering-tradeoffs that need to be "preferred" and optimized that particular application: dynamic versus static, performance versus memory, storage versus latency, throughput versus bandwidth, single versus multi processing, optimistic versus pessimistic concurrency, security versus availability, and on and on.

Software process reuse is no different. Different products and projects have their own uniquely differentiating value proposition (if not there would be no business-need to attempt them in the first place). And those differentiating aspects warrant many kinds of process variation across projects, products, technologies and teams.

Those coming from a SixSigma background may point out how SixSigma strives for reducing process variation. But it's all too easy to forget the context for that is when repeatably reproducing the same output from the same inputs for the same desired set of quality attributes and tradeoffs (not to mention the "kinds" of variation SigSigma is appropriate for trying to eliminate).

So I would advocate the translation and application of software product-line practices to software processes (software "Process-Lines" or "Process Families" if you will) and the treatment of such common processes as first class architectures that need to accommodate the views and perspectives of ALL their critical stakeholders, and which should identify their essential quality attributes and tradeoffs, approaches to managing commonality and variability, and apply appropriate patterns and tactics (such as modifiability tactics for software processes and projects) to meet those objectives.

In light of the above, Lean & Agile software development define an architectural style for such process families and their architecture. Lean and Agile principles identify some of the critical process-quality attributes for such efforts. And the corresponding enterprise and its product offerings and their market segments may identify additional quality attributes that needs to met (such as security, regulatory auditability/compliance, large-scale and/or distributed projects and teams, etc.)