Wednesday, June 24, 2009

Technical Debt - Definition and Resources

I ran across a few really good papers on the subject of technical debt that are fairly comprehensive in their treatment of not just what it is, but also how to manage it:
For those not already well-versed on the subject, Technical Debt [a.k.a. Design Debt] occurs when our software becomes difficult or risky to change, and takes increasingly more time & effort to evolve. Technical debt represents the cost of the accumulated amount of rework that will be necessary to correct and/or recover from the deviation between:
  • the current design of the system, versus ...
  • a design that is minimally complex yet sufficiently complete to ensure correctness & consistency for timely delivery.
This effort grows more than linearly over time as a system becomes bigger and more complex.

The economic impact of technical debt is directly related to the cost of complexity and its resulting “friction” against the velocity of the development team (and, subsequently, upon the ease of system evolution).

Technical debt can be caused by under-engineering just as much as it can be caused by overengineering (overdesigning). It is a difficult, delicate and dynamic balancing act to achieve the necessary and sufficient amount of design to implement only the essential complexity required by system:
  • Sometimes we knowingly (under great pressure) do something the "quick & dirty" way, with the intent to "do it right" later (but not too late).
  • Sometimes we try to do too much to soon, and elaborate or implement details of the requirements when we and our customers/users haven't yet learned enough about the true needs of the system.
  • Sometimes we haven't yet learned enough about good design, and unintentionally violate design principles, resulting in undesirable dependencies that make code or other work-products hard to change.
  • Sometimes we neglect to properly "tend" to the design and don't give it the necessary amount of ongoing care and feeding needed to keep it "fit" and "supple."
But whether it is a deviation between principles and practice (knowingly or unknowingly), guessing incorrectly, anticipating too early, neglect, poor quality, or even just the laws of software evolution, we must make plans and take action to deal with the inevitable entropy of evolution or else we will sink too far into technical debt.

That's just my two cents on the subject of course. What do others have to say about it?

Wikipedia defines Technical Debt by referring to the words of Ward Cunningham, who first drew the comparison between technical complexity and debt in a 1992 experience report:
"Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."

On the C2 Wiki, Ward defined Technical Debt as:
“Technical Debt includes those internal things that you choose not to do now, but which will impede future development if left undone. This includes deferred refactoring. Technical Debt doesn't include deferred functionality, except possibly in edge cases where delivered functionality is "good enough" for the customer, but doesn't satisfy some standard (e.g., a UI element that isn't fully compliant with some UI standard).”

Martin Fowler's definition of Technical Debt is also frequently cited:
"Doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future."

In his introduction to Refactoring to Patterns, Joshua Kerievsky quite simply defines design debt as follows:
Design debt occurs when you don't consistently do three things.
  1. Remove duplication.
  2. Simplify your code.
  3. Clarify you code's intent.
Few systems remain completely free of design debt. Wired as we are, humans just don't write perfect code the first time around. We naturally accumulate design debt ....

Due to ignorance or a commitment to "not fix what ain't broken," many programmers and teams spend little time paying down design debt.... In financial terms, if you don't pay off a debt, you incur late fees. If you don't pay your late fees, you incur higher late fees. The more you don't pay, the worse your fees and payments become. Compound interest kicks in, and as time goes on, getting out of debt becomes an impossible dream. So it is with design debt.

James Shore describes Design Debt in much the same manner:
When the cost of change increases, it's because the design quality is decreasing. Since retiring or rewriting software means writing off a huge investment, you wouldn't expect any team to let design quality deteriorate to that point. Yet it happens all the time. Why?

Design debt" explains the problem. When a team is working under pressure, they take shortcuts that compromise design quality. It's like taking out a high-interest loan. The team gets a short-term boost in speed, but from that point forward, changes are more expensive: they're paying interest on the loan. The only way to stop paying interest is to pay back the loan's principle and fix the design shortcuts.

Steve McConnell defines Technical Debt as follows:
“Technical Debt” refers to delayed technical work that is incurred when technical short cuts are taken, usually in pursuit of calendar-driven software schedules. Just like financial debt, some technical debts can serve valuable business purposes. Other technical debts are simply counterproductive. The ability to take on debt safely, track their debt, manage their debt, and pay down their debt varies among different organizations. Explicit decision making before taking on debt and more explicit tracking of debt are advised.

Mary Poppendieck gives a definition of Technical Debt in her upcoming book Leading Lean Software Development:
“All successful software gets changed. So if we think we’re working on code that will be successful … we need to keep it easy to change. Anything that makes code difficult to change is technical debt. Just like any other debt, the cost of paying off technical debt gets more and more expensive over time. … Technical debt drives the total cost of software ownership relentlessly higher … eventually we will have to pay it off or the system will go bankrupt.”

Kane Mar, in Technical Debt and Design Death, describes technical debt and later likens it to the effects of entropy:
Technical debt is simply defined as deferred work that is not directly related to new functionality, but necessary for the overall quality of the system. Examples of this include delaying upgrades to necessary tools and frameworks, delaying the refactoring of overly complex system components, and so on. If this technical work is necessary for the health of the system, then what happens when it is ignored for too long? What happens when several layers of cruft accumulate and there are no unit tests to aid refactoring? ...

In thermodynamics, “entropy” refers to the randomness of the components of a system. When the term is applied to software it is considered a measure of disorder. Thus entropy in software is the result of changes made to the code base, including bug fixes, updates to existing functionality, and the addition of new functionality. But over a period of time, these small changes can snowball to create a system that is difficult to change, overly connected to external systems, and lacks clear delineation of functionality.

Chris Sterling, in his upcoming book on Architecture in an Agile Organization, distinguishes between Software Debt and Technical Debt:
Software debt accumulates when focus remains on immediate completion while neglecting changeability of the system. The accumulation of debt does not impact software delivery immediately. At first it creates a sense of increased feature delivery with management, business stakeholders and the team. Business stakeholders respond well to the pace of delivered functionality. What they don’t understand is that this only represents an illusion of earlier returns on their investment.
...
This allows both the business and software delivery teams to live in the illusion of status quo far longer than they should. At some point previously small forms of decay in the system become large enough to affect our software delivery to the point that working harder and longer doesn’t result in success.
...
Debt is made glaringly visible when the team works on stabilizing the software functionality late in the release cycle. Integration, testing, and bug fixing is unpredictable and does not get resolved adequately before the release. People involved in the project stay late working to get the release out the door. It is now too late to pay back the debt accrued during the feature development.

The following sources constitute what I call software debt:
  • Technical Debt: those activities that a team or team members chooses not to do now and will impede future development if left undone
  • Quality Debt: diminishing ability to verify functional and technical quality of entire system
  • Configuration Management Debt: integration and release management become more risky, complex, and error-prone
  • Design Debt: cost of adding average sized features is increasing to more than writing from scratch
  • Platform Experience Debt: availability and cost of people to work on system features are becoming limited

Over at the Agile-in-a-Flash blog, Jeff Langr and Tim Ottinger provide a flashcard of tips & truths about technical debt, along with a more detailed discussion about its nature and origins. They distill several nuggets of wisdom like:
  • Incurring technical debt means your velocity slows down and you will deliver less value
  • The cost of getting out-of-debt is compounded over time: the longer you wait, the faster it grow!
  • If you plan to incur technical debt, the persons responsible must have a workable plan to pay it off!
  • “Interest only” payments won’t improve things
  • Pay early, pay often, and pay-as-you-go. (The only other options are bankruptcy or death.)
  • Remember: Those with the worst debt problems often have the most difficulty imagining a life without borrowing!

So how might we properly measure/monitor and account for technical debt? Well, some of the above papers I mentioned have some good ideas. Some that address it more explicitly are:
Here are a number of other very good resources on technical debt culled from blogs, articles, and presentations over the years. Some of them describe it, while others delve into methods for managing it and paying it off:
Posted by at
Labels: , ,

3 comments:

Anonymous said...

This is a great post with a ton of highly useful reference materials for further research.

This is a topic that I've been coming to grips with over the past few years and good to see that the community and resources are starting to gain traction.

We've recently begun trying to take action to reduce technical debt within my organization. Just using some rudimentary measurement techniques.

1:12 PM
Steven M Smith said...

Brad, Terrific post! Thank you. -Steve

11:37 AM
Anonymous said...

I'm in the middle of a 6 part series on technical debt and wish I would have come across your list of resources before I started. Based on my experience, measuring the magnitude of the debt is the most challenging aspect.

http://blog.acrowire.com/technical-debt/technical-debt-part-3-quantifying/

11:29 AM

Post a Comment

Subscribe to: Post Comments (Atom)