Monday, July 06, 2009

JEDI Programming - Just Enough Design Initially

I left a comment on the "What is Missing?" entry at the Agile-in-a-Flash blog. The author's asked the questioin "What is missing?" from the stack of Agile flashcards they are developing. I responded ...


I think the "JEDI" approach is missing (any by that, I don't mean the mantra of "use the source Luke" ;-)

I think there is something missing regarding TDD and Design. Uncle Bob's three rules of TDD (and other writings) often mislead people to think that there is ZERO design up-front, as is if NOT doing Big-Design-Up-Front (BDUF) implies that therefore there is zero up-front design (NoDUF).

This is false (and Uncle Bob has even vehemently said so in The Scatology of Agile Architecture) but how does a newcomer reconcile it with the three rules of TDD? I can't write test-code without being able to invoke the thing-under-test. I can't invoke a thing if I haven't attempted to design the interface.

If I design an interface (even for a single method/subroutine) I have to have some inkling of which class/package/module it would go in, at least INITIALLY! There is some initial amount of design I do before writing a test that is both necessary and sufficient to define "just enough" of the interface of what I want my test-case to test.

So I think that is what is missing, a card called "JEDI", for "Just Enough Design Initially."

To my knowledge, this particular definition of the JEDI acronym in agile development was first used by Stephen Palmer and other FDD luminaries at www.step-10.com and featuredrivendevelopment.com (just do a Google-search on "JEDI" AND "Just Enough Design").

I also think there is a relationship between JEDI and Eric Evan's " Domain-Driven Design (DDD), Supple Design (part of DDD), as well as *some* of the so-called "Pre-factoring". But it can be a risky, slippery-slope, so it would be great to have some guidance to help us know when we've done "Just Enough Design In-front/Initially."

I suppose JEDI is a way of straddling the "appropriate range" of risk between anticipation and adaptation. I envision some kind of graph or diagram with axes ...
  • On the left-hand side of overanticipating we have "too much too soon" and big/all up-front design.

  • All the way on the right-hand-side we have "too little too late." Here you are faced with legacy-rewrites, system re-engineering, large-scale restructuring, etc.
The problem with both extremes is the creation of technical debt:
  • One Extreme does it by adding complex structures and behaviors too early in the cone-of-uncertainty and causes too much rework to rewrite that which was not yet certain and subject to much variability.

  • The other extreme does it by sheer entropy/decay/rot that results from inattention and/or negligence
In the middle we have a delicate balance that we need to strike, and which JEDI represents: "Just enough" and "Just-in-Time." The problem is that this is a range, not a single perfect point. What makes the problem harder is that the range is different for different projects, depending on a number of factors (including scale and distribution). There is a certain amount of "pay-up-front" and "pay-later" that need to be balanced with "pay-as-you-go."

I imagine this "range" represents how to find the "sweet spot" that demarcates Just Enough Design Initially:
  • In the middle of the scale is pure refactoring. It is strictly emergent, pay as you go just-in-time by focusing ruthlessly on keeping code clean and simple.

  • Minor-to-moderate restructurings (too large to be refactorings, too small to be total re-writes) are to the right. Sometimes larger systems and/or systems constructed by multiple-teams can do a great job at refactoring and still not be able to avoid the need for minor-to-moderate restructuring, particular for aspects of the design that cut across teams and architecture and functionality.

  • Just to the left of refactoring would be so-called "pre-factoring", where you have enough experience refactoring that you are able to apply basic application of encapsulation, DRY, separation of concerns, etc. without adding premature abstraction of inessential complexity. This is hard to get right, and has risk associated with it. But it does get progressively less with the better judgment that comes experience and practice.

  • To the left of pre-factoring would be the subset of DDD known as Supple Design. And the rest of DDD would be to the left of supple-design.
Somewhere in this continuum might be "segments" corresponding to specific practices or techniques besides DDD and refactoring, such as clean code, simple/incremental design, and evolutionary architecture.

The trick of knowing the "just enough" range lies not just in experience and discipline, but also in understanding your context and your "lead-time". Lead-time in particular dictates how soon in advance you need to be able to think and anticipate (and at what level of detail). The shorter the lead/cycle-time, the less you need to anticipate and prognosticate.

So I would see JEDI as a card that somehow is able to depict this continuum between anticipation and adaptation and where these various techniques fall on it, and the factors/tradeoffs that help you identify the right "range" for yourself. (And of course there is a close relation to technical debt and the cost-of-change curve :-)

Anyone ever seen a diagram that bears any resemblance to what I'm thinking of here?

3 comments:

Unknown said...

Hi Brad,

I think that there's value in recognizing the historical context of pre-factoring and not fail to recollect that it was largely an opposition to emergent design rooted deeply in not having the firmest grasp on the meaning of refactoring at the time.

This article seems to use "pre-factoring" as a reflection of "refactoring". Almost every appearance of "pre-factoring" could be replaced with "design".

I don't particularly see pre-factoring as synonymous with design because pre-factoring is a response to refactoring.

It might seem logical that pre-factoring is the opposite of refactoring, but there is no opposite of refactoring, as refactoring is also just design. No design or no maintenance of design might be the opposite of factoring and refactoring.

When the factoring turns out to be imprecise or no longer appropriate, we redo it: refactoring.

Whether the factoring is done far in advance of proving its applicability to reality or if its done just before implementation, it's still just factoring, and factoring is just design.

Brad Appleton said...

Hi Scott! Thanks for your comment. While I agree that prefactoring, as a term, came about partly as a form of resistance to "emergent design", I think that misunderstanding from which it arose is a common one, and the same one that Uncle Bob complains about so strongly in his "Scatology of Agile Architecture" posting.

This is a misunderstanding that is all too easy and all too common to have, precisely because "JEDI" is not explicitly emphasized (certainly not enough) in the dissemination of most information about refactoring, TDD and emergent design.

This article uses the definition of refactoring that Ken Pugh used in his article that attempted to clarify what it meant from the hype and marketing associated with the book (It was not quite so extreme a counter-reaction to emergent design as many had thought).

While refactoring and prefactoring are both forms of factoring which are in turn forms of design, they carry a specialized enough meaning that is more specific in scope and timing as to not simply be synonomous with "design".

Therefore, it has its place on any spectrum that compares the time/timing of the design that is done along with the amount (big vs small, too soon vs too late, and everything else on both sides of the "just enough & just in time" spectrum.

I was actually kind of surprised that of everything said in the article, you seemed to focus exclusively on this one small aspect of it. Is "prefactoring" a "hot button" that is the cause of bad blood/feelings by many in the agile community?

Brad Appleton said...

Just a follow-up (albeit late), at Agile 2009 I attended a session by Neal Ford on "Evolutionary Architecture and Emergent Design" which in turn is based on material from his book "The Productive Programmer"

Neal also has a series of articles on the topic @ developerWorks.com and the introductory article has a graph of sorts that I think starts to get close to what I attempted to describe above for JEDI (see Figure 1 "Spectrum of Design" in the section titled "Defining Design" at http://www.ibm.com/developerworks/java/library/j-eaed1/

Basically the horizontal axis of the timeline shows how much before or after you do the design: all-up-front, incrementally up-front, immediately before (test-first), immediately after (as in refactoring), incrementally after (restructuring), rewriting/reengineering etc.

Peppered about that timeline are techniques or practices such as evolutionary architecture, domain-driven-design, incremental design, TDD/BDD, refactoring, restructuring, etc. Each technique tries to place itself in the correct horizontal position on the timeline.

Then "JEDI" itself is basically a "curve" across this horizontal axis (time) and vertical axis (size/complexity) that looks a bit like a bell-curve of sorts, the area under which indicates how much up-front, during, and after-the-fact design is appropriate. Projects of different size/complexity would have slightly different curves acknowledging the need for more up-front work.

The area under the curve would correlate somewhat to technical-debt. Meaning the more you design too-early, the more technical debt you are creating for later, and the more you wait until its too-little-too-late the more you are accumulating without "paying down" your debt.

And that would be the JEDI card. Its sort of like cost-of-change curve accept it is the cost of designing too-much-too-soon versus too-little-too-late with "JEDI" indicating the "sweet spot" (for a projects size/complexity and risk-threshhold [e.g., spaceshuttle launches])

The "techniques" that fall on the appropriate points in the horizontal timeline would likely also correspond to other cards in the tech (TDD, BDD, DDD, Refactoring, Restructuring, Emergent Design, Incremental Design, etc.)