Sunday, July 26, 2009

Resources on Retrospectives

I found a really good resource-list from George Dinwiddie on Introspection and Retrospectives that includes the following list of resources (mostly patterns & techniques) about conducting retrospectives. It contains many (but not all) of the links below:
Posted by at No comments:
Labels: ,

Friday, July 17, 2009

Refactoring @ Scale

In my previous post, Refactoring for Agility, I posted an outline and some thoughts for Part I of an Overview on Refactoring. Now I'm ready to post on Part II which is about refactoring @ scale. By "at scale" I mean in the larger context of other agile practices, as well as for large projects.

PART II - REFACTORING @ SCALE

1. Scaling-Up
  • To scale refactoring for larger projects, some additional techniques & issues must be added to the mix.
  • Note that this is “in addition to” (not “instead of”)
    Refactoring In-the-SmallRefactoring @ Scale
    Small, Fast & Frequent RefactoringsLarger, Periodic & Planned Restructurings
    Emergent DesignIncremental Design & Evolutionary Architecture
    Deferred RefactoringRestructuring & Technical Debt
    Code SmellsArchitecture Smells
    Design Principles & PatternsSoftware Modifiability Tactics
    Simple/Clean CodeSupple/Domain-Driven Design

2. Emergent Design
  • Emergent Design is a fancy name for the resulting design that “emerges” from the synergy of combining Refactoring together with TDD, Continuous Integration and Automated Testing.
  • Emergent Design means that ...
3. Technical Debt [a.k.a. Design Debt]
4. Restructuring Technical Debt
  • If we accrue a non-trivial amount of technical debt, we can’t simply “refactor” it away.
  • Paying it off typically requires restructuring efforts (or even reengineering) that must be planned.
  • Iteration plans must accommodate specific tasks for these restructuring efforts (or even be dedicated to restructuring).
  • Ignoring it, or deferring it for very long is not a viable option!
5. Overview of Restructuring
  • Identifies higher-level issues (“architecture smells”) that typically represent violations of known principles of good software architecture & design.
  • Periodically applies larger-scale “refactorings” and/or many small refactorings that were previously deferred.
  • The goal is to “pay down technical debt” in order to limit the increasing costs of accumulated complexity.
  • Typically requires a concerted effort that must be separately planned.
  • Uses not only design patterns/principles, but also architectural patterns/principles, as well as software modifiability tactics.
6. Refactoring vs. Restructuring
7. Restructure Periodically
  • Restructuring is often associated with absent or neglectful refactoring and/or design.
  • But … Any large software project spanning multiple teams eventually needs restructuring.
    • Even in the presence expert-level architecture, design & continuous refactoring
    • This is just a reality of software evolution/entropy
  • Therefore … Large software projects should assume that periodic restructuring will be necessary, and should plan accordingly to:
    • Clean-up accumulated code-smells and apply numerous refactorings that were deferred but are now sorely needed,
    • Address architecture smells by applying restructurings, patterns and modifiability tactics that have broader impact.
8. Architecture Smells
See Stefan Roock's and Martin Lippert's book Refactoring in Large Software Projects (There was an earlier version of their work entitled Large Refactorings").
  • Smells in Dependency Graphs
  • Smells in Inheritance Hierarchies
  • Smells in Packages
  • Smells in Subsystems
  • Smells in Layers
9. Software Modifiability Tactics
  • Localize Changes (increase cohesion)
  • Prevent Ripple Effects (reduce coupling)
  • Defer Binding-time (defer decision-making)
10. When to Consider/Plan Restructuring?
11. Evolutionary Architecture
  • Software Architecture concerns infrastructure elements that must exist before you can begin execution.
    • Architecture is about things that are hard to change later, it is difficult to allow an architecture to emerge.
    • For large projects, this includes high-level organization of the system into functionality/elements that will be allocated to separate teams.
  • Key techniques of Evolutionary Architecture include:
    • Deferring Irreversible Decisions to the “Last Responsible Moment” (LRM Principle)
    • Architectural “Spike”, Architecture Iteration and/or Spike Iteration
12. Incremental Design
  • Design evolves incrementally, iteration by iteration, based on current business priorities and discovered technical limitations.
  • Incremental Design …
    • Does not prohibit thinking about higher-level design.
    • Does encourage planning in detail only what will be constructed soon.
  • Focus is on “Just Enough, Just-In-Time” :
    • Specifying too much detail too soon causes more rework later.
    • But doing less now and saving the rest for later should not require significantly more work later than it would today.
  • We must do Just Enough Design Initially to attain the right balance of anticipation and adaptation.
13. Just Enough Design Initially (JEDI)
  • Initial design (before coding) is still necessary.
    • This use of “JEDI” was coined by Stephen Palmer as part of Feature-Driven Development (FDD)
  • Basic rule of thumb to tell when “JEDI” is achieved:
    • At iteration-scope, when, after one pass through the iteration backlog, modeling in small groups does not produce any new classes or associations of real significance.
    • At task/TDD scope, when we have defined enough structure & interface(s) to know specifically what code to write/test, precisely where to write it, and exactly how to invoke it.
  • Techniques of “the JEDI way” include:
    • Collaborative Domain Modeling and Color Modeling [from FDD]
    • Supple Design techniques & patterns
    • Domain-Driven Design (a.k.a. DDD -- see domaindrivendesign.org)
    • Design Blitz & other Agile Modeling techniques (see agilemodeling.com)
14. Supple Design & DDD
  • Domain-Driven Design (DDD) approaches modeling the core logic of the software by focusing on the domain.
    • The basic idea is the design should directly reflect the core business domain and domain-logic of the problem to solve
    • This helps understanding the problem as well as the implementation and increases maintainability of the software.
  • DDD uses common principles and patterns as "building blocks" to model & create a "supple design”
    • Supple: pliant, malleable, limber, yielding or changing readily.
    • The design is firm yet flexible, with structure and intent both clearly conveyed and deeply realized by the code.
  • Patterns of Supple Design include: Intention-Revealing Interfaces, Ubiquitous Language, Side-Effect-Free Functions, Assertions, Conceptual Contours, Standalone Classes, Closure of Operations, Declarative Style

15. Resources:

Resources on Emergent Design and Evolutionary Architecture Resources on Restructuring Resources on Technical Debt Resources on Modifiability Tactics Resources on Supple Design & DDD
Posted by at No comments:
Labels: ,

Thursday, July 16, 2009

Refactoring for Agility

Some of you might have guessed from my recent posts on Emergent Design, Technical Debt, JEDI Programming, and 5S Qualities of Well Designed, Well-Factored Code, that I've been looking into trying to teach the fundamentals of refactoring and how it scales to larger projects. I've gathered some references and quotes and some ideas for slides that I wanted to bounce around on my blog.

Here is an outline and some thoughts for part I of some slides ...

PART I - REFACTORING FOR AGILITY

1. Overview of Refactoring
  • Identifies design-maintenance issues (“code smells”) that typically represent violations of known principles of good design.
  • Incrementally and iteratively applies a set of design improvement techniques (“refactorings”).
  • The goal is to minimize complexity & duplication in order to maximize simplicity & ease-of-change.
  • Encourages the “right” design details to emerge “just-in-time” with minimal guesswork/rework.
  • Scaling-up includes the use of periodic restructuring, initial & incremental design (“just enough”), and evolutionary architecture.

2. Refactoring Defined [cite definition(s)]

3. Refactoring Myths -- Refactoring is NOT …
  • “Rework” – redesigning things that could, and should, have been designed correctly in the first place.
  • Design “gold-plating” – work that adds no business value, and merely serves to stroke the egos of perfectionists who are out of touch with business reality.
  • Miscellaneous code “tidying” – the kind that is “nice to have,” but should only happen when the team has some slack-time, and is a luxury we can do without, without any serious consequences.
  • A license to “hack” – avoiding any and all initial design & analysis and instead jumping straight to coding with no “real” design.
  • Reengineering – large-scale restructuring that requires a concerted effort over the course of several weeks/months to re-write or re-architect significant parts of the system.
4. Refactoring IS …
  • A systematic approach to source-code “hygiene” that minimizes the chances of introducing bugs
  • Improving the design of the code after it has been written
  • A behavior-preserving transformation of source-code structure
  • The process of simplifying & consolidating a work-product by making several, small, successive revisions focused on: preserving correctness, removing redundancy, revealing thoughts & intentions, and improving clarity & conciseness.
  • A disciplined way of making changes while exposing the project to significantly less risk.
  • An effective means to address the economic reality of software growth/complexity by reducing & amortizing its cost throughout the daily business of code development & maintenance activities.
5. Why Refactor?
6. How to Refactor
7. Rules of Clean Code
8. Rules for Simple Code
9. The Steps of Refactoring
10. Code Smells
11. Categories of Refactorings
  • Small Refactorings
  • Larger Refactorings/Restructurings
  • Each category contains as many as a dozen or more refactorings, most of which are catalogued at http://refactoring.com/catalog/

12. Refactorings (Some refactorings from real projects)
  • See http://refactoring.com/catalog/ for an up-to-date list (and the “Refactoring to Patterns” catalog too)
13. What to do if …?
  • I spot a “smell” that is not already known or catalogued?
  • There is no specific known/catalogued “refactoring” for what I think I need?
14. When to Refactor?
  • While adding functionality
  • While fixing a bug
  • While reviewing code
  • After coding the same/similar thing for the third time (to “factor out” the duplication)
  • A.k.a.: The Rule of Three: 3 strikes and you refactor.
  • After the third time you deferred refactoring a change, for any reason [The Rule of Three, again]
  • Before the end of the iteration if you haven’t been following The Rule of Three
15. Refactor Continually
16. When NOT to Refactor?
  • When the build is broken or tests don’t pass
  • When it would compromise meeting an impending deadline or commitment
  • When the code in question really just needs to be re-written “from scratch”
  • When it would modify code/interfaces that could significantly impact/break other work (e.g.: Published/public interfaces and protocols, Database schemas/tables/operations)
  • Sometimes we must defer refactoring for later and/or plan for subsequent restructuring
17. Refactoring to Patterns & Principles
Software Design Principles and Design Patterns are the underlying foundation for Refactoring:
  • Code smells (a.k.a “code pathologies”): Signal a possible violation of design principles, Suggest which refactoring may be needed
  • Refactoring: Correct a design principle violation (at least partially), Converge toward common design patterns
  • Design Patterns: Reconcile forces among conflicting design concerns,Restore balance between competing design principles
  • Design Principles: Lead us to attain desired design qualities/attributes
18. Design Attributes/Code Qualities
Qualities of Highly Maintainable Software:
  • Loose Coupling & High Cohesion
  • Hierarchy (Structural Decomposition)
  • Abstraction, Encapsulation & Modularity
  • Sufficiency, Parsimony and Primitiveness
  • Readability
  • Testability
  • Modifiability
  • Serviceability
19. Design Principles: SOLID, SoC, DRY, Shy
  • The SOLID Principles of Object-Oriented Design (from Uncle Bob)
  • The SoC Principle: Separation of Concerns — separate interface from implementation, policy from mechanism, behavior from construction, commands from queries, ...
  • The DRY Principle: Don’t Repeat Yourself (Eliminate Duplication), Single Point of Truth
  • The “Structure-Shy” Principle: (“Tell, Don’t Ask!”), The Law of Demeter, Principle of Least Assumed Knowledge
20. Other Acronyms of Simple/Agile Design
  • OAOO – Say Things Once And Only Once (restatement of the DRY principle)
  • DTSTTCPW – Do The Simplest Thing That Could Possibly Work! (restatement of the KISS principle)
  • YAGNI – You Aren’t Gonna Need It!
  • The LRM Principle: Defer Commitment of Irreversible Decisions to the Last Responsible Moment!
  • BDUF – Big Design Up-Front! (vs. JEDI)
  • JEDI – Just Enough Design Initially/In-front!
  • DDD – Domain-Driven Design
21. Design Patterns
22. Summary: Refactoring for Agility
  • Successively applies small behavior-preserving transformations to eliminate code smells
  • Based on proven design principles and patterns for achieving maintainability & modifiability
  • Good automated testing is a prerequisite
  • Refactoring is not rewriting, rework or restructuring
  • With refactoring, we continuously invest nominal effort to reduce the risk & cycle-time of changes
  • The goal is to minimize complexity & duplication in order to maximize simplicity & ease-of-change.
  • Practiced in a highly disciplined manner, it promotes:
    • Sufficient functionality
    • Simple & clean code
    • Supple design
    • Serviceable software
    • Sustainable team velocity

23. Resources:
Code Smells:
Design Principles:
Design Patterns:
- Online Resources:
- Books:
Other Agile Design Slogans:
Posted by at No comments:
Labels: ,

Tuesday, July 14, 2009

Mercurial, Git and Scala

Three more brand new books I just received that are worth mentioning ...


For those who may not know ...
  • Mercurial (like Git) is an open-source, distributed version-control system.

  • While Scala is the "hot" new JVM-based programming language that not only contains the "best of both worlds" from other statically-typed languages (like Java) as well as dynamic languages like Groovy, but also the best of both worlds from object-oriented programming languages (OOPLs) and functional programming languages (like Erlang and Haskell). Plus it has message-passing and actor-based support for concurrency (like Erlang) while still being strongly typed and having access to all your favorite Java APIs (and anything else that compiles to JVM). Not too mention that Scala has a kick-butt Web development framework called Lift that is a next-generation framework to the likes of Rails/Grails and Apache Struts.

Scala is my first pick for the programming language that is most excitingly poised for the multi-core programming revolution in the age of multicore processors, cloud-computing, web 2.0 and access to all your favorite Java APIs/apps.
Posted by at No comments:
Labels: , ,

Saturday, July 11, 2009

BOOK: Landing the Tech Job You Love

I blogged earlier about The Passionate Programmer and The Nomadic Developer.

A new book just came out that seems like the perfect complement to these two: Landing the Tech Job You Love by Andy Lester (also from the Pragmatic Programmers). I've only just started reading it but so far I like it a LOT! It also received a nice review by Mike Riley in DDJ online.

Rather than cite the blurb from the book this time, I rather like the description given in the press release:

It’s tougher than ever to get that great job. Companies are more demanding and your competition is smart, tech-savvy, and resourceful. You’ve got the right skills for the job—you also need the right skills for job hunting. You need to work deliberately on your new job hunting skills, and this book can help. Old fashioned cookie-cutter job hunting skills aren’t enough: Land the Tech Job You Love gives you the background and the hard-won wisdom to leapfrog those who play by the old rules.

Andy tells us, “Life is too short for a job you don’t love. You’re not stuck—other opportunities are available for you, if you know where to look and can work the hiring process to your advantage. This book will help you get that job you love.”

In this book, you’ll learn how to find the job you want that fits you and your employer. You’ll uncover the hidden jobs that never make it into the classifieds or Monster. You’ll start making and maintaining the connections that will drive your future career moves.

Andy started writing this book years before the recession (a.k.a. “econopocalypse”) hit. He looked at the conventional wisdom and the advice available in generic books on job hunting, and found the conventional wisdom just didn’t work for programmers, system administrators, testers, and other related development positions.

He looked at everything from whether you should look for work on online job boards to whether you should lead off your resume with your objectives. Although he has definite answers for these two, he found that the answer to most questions is “it depends.” His book leads you to taking an honest assessment of what you offer and what you want in a job so that you end up in a job that is a good fit for you and your employer.

This is an important book for you to read whether you currently have a job or not. The same tactics you take to make yourself more employable will also make it easier to get promoted in your current company.


Also see Andy Lester's presentation on Effective Job Interviewing from Both Side of the Desk and his blog @ TheWorkingGeek.com for his thoughts on "Job hunting and working life for programmers, sysadmins and all other techies"
Posted by at 1 comment:
Labels:

Wednesday, July 08, 2009

BOOK: The Economics of Iterative Software Development

In the July issue of the Agile Journal I reviewed Walker Royce, Kurt Bittner and Mike Perrow's book The Economics of Iterative Software Development: Steering Toward Better Business Results. Here is an excerpt ...

The Economics of Iterative Software Development: Steering Toward Better Business Results is an important text for anyone trying to persuade management to "go iterative" as well as to anyone needing to measure & track the kinds of business results that management needs to see for a software development project.

I'll be perfectly honest: I was expecting this book to be an extremely dry and boring read, albeit full of lots of useful information densely packed in mathematical models and formulas, perhaps reminiscent of past college days reading a huge tome on socio-political economic theories. It wasn't as bad as I'd feared. Yes - it is a bit dry in places, but those are the exception rather than the rule. And there are lots of "war stories" sprinkled throughout that hold your interest and stop your eyes from glazing over.

...

All in all, The Economics of Iterative Software Development is a solid-book that is a relatively light/quick read with lots of good, practical advice on how to apply an economic model of thinking and measurement for managing and tracking business-results of an iterative project. If you need help doing that, or with communicating to your management about the need and benefits of an iterative approach, then you need to read this book!

See the full review for more details.
Posted by at No comments:
Labels: ,

Monday, July 06, 2009

JEDI Programming - Just Enough Design Initially

I left a comment on the "What is Missing?" entry at the Agile-in-a-Flash blog. The author's asked the questioin "What is missing?" from the stack of Agile flashcards they are developing. I responded ...


I think the "JEDI" approach is missing (any by that, I don't mean the mantra of "use the source Luke" ;-)

I think there is something missing regarding TDD and Design. Uncle Bob's three rules of TDD (and other writings) often mislead people to think that there is ZERO design up-front, as is if NOT doing Big-Design-Up-Front (BDUF) implies that therefore there is zero up-front design (NoDUF).

This is false (and Uncle Bob has even vehemently said so in The Scatology of Agile Architecture) but how does a newcomer reconcile it with the three rules of TDD? I can't write test-code without being able to invoke the thing-under-test. I can't invoke a thing if I haven't attempted to design the interface.

If I design an interface (even for a single method/subroutine) I have to have some inkling of which class/package/module it would go in, at least INITIALLY! There is some initial amount of design I do before writing a test that is both necessary and sufficient to define "just enough" of the interface of what I want my test-case to test.

So I think that is what is missing, a card called "JEDI", for "Just Enough Design Initially."

To my knowledge, this particular definition of the JEDI acronym in agile development was first used by Stephen Palmer and other FDD luminaries at www.step-10.com and featuredrivendevelopment.com (just do a Google-search on "JEDI" AND "Just Enough Design").

I also think there is a relationship between JEDI and Eric Evan's " Domain-Driven Design (DDD), Supple Design (part of DDD), as well as *some* of the so-called "Pre-factoring". But it can be a risky, slippery-slope, so it would be great to have some guidance to help us know when we've done "Just Enough Design In-front/Initially."

I suppose JEDI is a way of straddling the "appropriate range" of risk between anticipation and adaptation. I envision some kind of graph or diagram with axes ...
  • On the left-hand side of overanticipating we have "too much too soon" and big/all up-front design.

  • All the way on the right-hand-side we have "too little too late." Here you are faced with legacy-rewrites, system re-engineering, large-scale restructuring, etc.
The problem with both extremes is the creation of technical debt:
  • One Extreme does it by adding complex structures and behaviors too early in the cone-of-uncertainty and causes too much rework to rewrite that which was not yet certain and subject to much variability.

  • The other extreme does it by sheer entropy/decay/rot that results from inattention and/or negligence
In the middle we have a delicate balance that we need to strike, and which JEDI represents: "Just enough" and "Just-in-Time." The problem is that this is a range, not a single perfect point. What makes the problem harder is that the range is different for different projects, depending on a number of factors (including scale and distribution). There is a certain amount of "pay-up-front" and "pay-later" that need to be balanced with "pay-as-you-go."

I imagine this "range" represents how to find the "sweet spot" that demarcates Just Enough Design Initially:
  • In the middle of the scale is pure refactoring. It is strictly emergent, pay as you go just-in-time by focusing ruthlessly on keeping code clean and simple.

  • Minor-to-moderate restructurings (too large to be refactorings, too small to be total re-writes) are to the right. Sometimes larger systems and/or systems constructed by multiple-teams can do a great job at refactoring and still not be able to avoid the need for minor-to-moderate restructuring, particular for aspects of the design that cut across teams and architecture and functionality.

  • Just to the left of refactoring would be so-called "pre-factoring", where you have enough experience refactoring that you are able to apply basic application of encapsulation, DRY, separation of concerns, etc. without adding premature abstraction of inessential complexity. This is hard to get right, and has risk associated with it. But it does get progressively less with the better judgment that comes experience and practice.

  • To the left of pre-factoring would be the subset of DDD known as Supple Design. And the rest of DDD would be to the left of supple-design.
Somewhere in this continuum might be "segments" corresponding to specific practices or techniques besides DDD and refactoring, such as clean code, simple/incremental design, and evolutionary architecture.

The trick of knowing the "just enough" range lies not just in experience and discipline, but also in understanding your context and your "lead-time". Lead-time in particular dictates how soon in advance you need to be able to think and anticipate (and at what level of detail). The shorter the lead/cycle-time, the less you need to anticipate and prognosticate.

So I would see JEDI as a card that somehow is able to depict this continuum between anticipation and adaptation and where these various techniques fall on it, and the factors/tradeoffs that help you identify the right "range" for yourself. (And of course there is a close relation to technical debt and the cost-of-change curve :-)

Anyone ever seen a diagram that bears any resemblance to what I'm thinking of here?
Posted by at 3 comments:
Labels: ,

Thursday, July 02, 2009

Emergent Design and Evolutionary Architecture - Resources

As a bit of a follow-up to my earlier posting on Technical Debt - Definition and Resources I gathered some resources on the subject of Evolutionary Architecture and Emergent Design (which is closely related to refactoring, restructuring and reengineering).
Posted by at 3 comments:
Labels: , ,
Subscribe to: Posts (Atom)