Saturday, October 22, 2005

Bugs versus Enhancements

On the SCRUM Develoment Yahoo Group, Stephen Bobick initiated a discussion about Bugs versus Enhancements:
Here's something I've run into agile and non-agile projects alike: the blurring of distinction between bugs and enhancement requests. To me a bug is erroneous operation of the software based on the customer's requirements. That's fine when both sides agree to what the requirements are. Sometimes a bug can also be caused by a misunderstanding of the requirements by the team, however, and yes I'll still call this a bug. Often, however, customers will dub "missing" functionality (which was never discussed initially) or "nice-to-have" features, shortcuts and so on as "bugs"....

When I have tried to make the distinction between bugs and enhancements clearer to the PO or customer, sometimes through a SM, the customer thinks we are nit-picking, or trying to "play the blame game", rather than properly categorize and identify their feedback. One approach is to keep trying to educate and convince them anyways (on a case by case basis, if necessary). Another approach is just to let them call anything they want a "bug". Of course this can screw up your metrics (incidence of bugs) - something we are interested in at my current job (i.e. reducing the rate of new bugs and fixing bugs in the backlog).

Any words from the wise out in the trenches on how to best approach this? Obviously, with unit testing and other XP practices there is a claim that bug rates will be low. But if anything can be declared a bug, it becomes more difficult to make management and the customer believe the claims you make about your software development process and practices. And when this happens, the
typical response is to revert to "old ways" (heavy-handed, waterfall-type approaches with formal QA).

-- Stephen
I've actually had a lot of personal experience in this for the past several years. Here are some of the things I have learned...


The term "bug" and the term "defect" don't always mean the same thing:
  • Bug tends to refer to something "wrong" in the code (either due to nonconformance with design or requirements).

  • Defect often means something that is "wrong" in any work-product (including the requirements).

  • Hence, many consider ALL of incorrect, inconsistent, incomplete, or unclear requirements to be "defects": if they believe a requirement is"missing" or incorrectly interpreted, it's still a "bug" in their eyes.

  • Ive also seen some folks define "bug" as: anything that requires changing ONLY the code to make it work "as expected". If it requires a change to docs, the consider it a "change request" (and the issue ofwhether or not it is still a "defect" isnt really addressed)

  • Also, many folk's metrics (particularly waterfall-ish metrics for phase containment and/or screening, but I think also orthogonal-defect classification -- ODC) explicitly identify "missing requirements" as a kind of defect


If so, then be prepared to battle over the differences. Very often, the difference between them is just a matter of opinion, and the resolution will almost always boil down to a matter of which process (the bugfix process or the enhancement process) is most strongly desired for the particular issue, or else will become an SLA/contractual dispute. Then you can bid farewell to the validity of your defect metrics.

If your development process/practice is to treat "bugs" differently than "enhancements" (particularly if there is some contractual agreement/SLA on how soon/fast "bugs" are to be fixed and whether or not enhancements cost more $$$ but bugfixes are "free"), then definitions of what a bug/defect is will matter only to the extent outlined in the contract/SLA, and it will be in the customer's interest to regard any unmet expectation as a "bug".

If, on the other hand, you treat all customer reported "bugs" and "enhancements" sufficiently similar, then you will find many of the previous battles you used to have over what is a "bug" and what isn't will go away, and wont be as big of an issue. And you can instead focus on getting appropriate prioritization and scheduling of all such issues using the same methods.

If the customer learns that the way to get the thing they want when they want it is a matter of prioritization by them, and if the "cost" for enhancements versus bugfixes is the same or else isn't an issue, then they will learn that in order to get what they want, they don't have to claim its a bug, they just need to tell you how important it is to them with respect to everything else they have to prioritize for you.


None of the above (or any other) dickering over definitions is what really matters. What really matters is managing and meeting expectations. Sometimes business/organizational conditions mandate some contractual definition of defects versus enhancements and how each must be treated and their associated costs. If your project is under such conditions, then you may need to clearly define "bug" and "enhancement" and the expectations for each, as well as any agreed upon areas of "lattitude"

Other times, we don't have to have such formal contractual definitions. And in such cases, maybe you can treat enhancements and defects/bugs the same way (as noted earlier above).

Lastly, and most important of all, never forget that ...


If you can truly listen empathically and non-defensively (which isn't always easy), connecting with their needs at an emotional as well as intellectual level, and demonstrate that it is important to you, then EVERYONE becomes a whole lot easier to work with and that makes everything a whole lot easier to do.

Then it's no longer about what's a bug or what's an enhancement; and not even a matter of treating bugs all that differently from enhancements ... it simply becomes a matter of hearing, heeding and attending to their needs in a win-win fashion.

I'm sure there are lots of other lessons learned. Those were the ones that stuck with me the most. I've become pretty good at the first two, and have become competent at the third. I still need a LOT of work on that fourth one!!!


richard said...

I struggle with this in my software company.

We fix "bugs" for free as we guarantee our code to work to specification defined and agreed to by the client. I explain this in conversation as simply... If the specifications indicate that the system should add 2+2 and the delivered code shows a resulting value of anything but 4, it is a bug.
We charge $$ for "changes" which are changes to the specifications agreed to by the client. These are summed up as... "Yes, it should add 2+2, but in this case 2+2 should equal 5."
We charge for "enhancements" which are summed up as... "Yes, 2+2 does produce a result of 4, but we want the number 4 to be displayed larger, in red and flashing.

Dealing with clients that consider a bug as something that doesn't do what they want is frustrating and damaging to the reputation of your business.

While I side with my clients more than not, I find it simply amazing the number of clients who are shocked when we deliver exactly what we defined in our specifications.

I found this information useful.
Considering expectation, specification, and implementation, we arrive at these five scenarios:

Expectation = Specification = Implementation. No bugs. Many happy campers.
Expectation = Specification != Implementation. Error or defect to be corrected by developer/vendor.
Expectation = Implementation != Specification. Customer is happy with code-to-intent. Vendor has a requirements bug, to be addressed in the "as-built" product specification.
Expectation != Specification = Implementation. Customer is unhappy with code-to-specifications. Customer has a requirements bug, to be addressed by a change request or reset expectations.
Expectation != Specification != Implementation. Likely the result of ambiguous or undocumented requirement(s). General disagreement, unhappiness, and finger pointing.
Some closing comments:

Scenario 2 implies employing software development and QA practices for error preventation and detection, such as prebugging, inspections, and testing.
Scenario 4 may explain the increasing interest in methodologies relating to "Requirements Engineering" (or "Requirements Management") and "Domain Analysis". Unfortunately, this interest would be customer-driven (given the aforementioned scenario which favors the vendor/developer) and in itself, may present little quantitative (risk or cost reduction) incentive for the vendor/developer to add process overhead.
Scenario 5 is somewhat rare (i.e., breakdown in communications and understanding).

Brad Appleton said...

Thanks Richard! I REALLY LIKE your analysis matrix of expectation, specification and implementation.