Sunday, June 28, 2009

Embracing Change - quotable quotes on change and uncertainty

Every now and then I come across a good quote about embracing the fact that change and uncertainty are an essential inherent reality of software development. Here are the ones I like so far. Do you have another one you like? If so, leave a comment and share it with me.
    It’s inevitable that requirements will change. Business needs evolve, new users or markets are identified, business rules and government regulations are revised, and operating environments change over time. In addition, the business need becomes clearer as the key stakeholders become better educated about what their true needs are.
    -- Karl Wiegers, Cosmic Truths about Software Requirements
    The growing unpredictability of the future is one of the most challenging aspects of the new economy:
    • Turbulence—in both business and technology—causes change, which can be viewed either as a threat to be guarded against or as an opportunity to be embraced.
    • Rather than resist change, the agile approach strives to accommodate it as easily and efficiently as possible, while maintaining an awareness of its consequences.
    Facilitating change is more effective than attempting to prevent it.
    • Learn to trust in your ability to respond to unpredictable events; it's more important than trusting in your ability to plan for disaster.
    • Agile processes harness change for the customer's competitive advantage.
    -- Martin Fowler and James Highsmith, On the Agile Manifesto
    Orders must change rapidly in response to change in circumstances. If one sticks to the idea that once set, a plan should not be changed, a business cannot exist for long.
    -- Taiichi Ohno, Toyota Production System
    It’s not the strongest who survive, nor the most intelligent, but the ones most adaptable to change.
    -- attributed to Charles Darwin
    Doubt is not a pleasant condition, but certainty is absurd.
    -- Voltaire
    It is not necessary to change; survival is not mandatory.
    -- W. Edwards Deming
    When the rate of change outside exceeds the rate of change inside, the end is in sight.
    -- Jack Welch
    The Futility of Fighting Change: Requirements, technologies, teams, priorities, companies, usage, users, everything will change. There are things you cannot know until you know them. Give up. Change is inevitable. Our process must be a process, therefore, of change.
    -- Scott L. Bain The Nature of Software Development, May 2008
    Uncertainty is inherent and inevitable in software development processes and products.
    -- H. Ziv and D.J. Richardson, The Uncertainty Principle in Software Engineering, Aug 1996
    It will be important to organize future projects with enough agility to be able to adapt to the additional emergence of new sources of unpredictable change.
    -- Barry Boehm, Making a Difference in the Software Industry, March 2008
    • When we program, we transform a poorly understood problem into a precise set of instructions that can be executed by a computer.
    • When we think we understand a program?s requirements, we are invariably wrong.
    • When we do not completely understand a problem, we must research it until we know that we understand it.
    • Only when we truly understand a problem can we develop a superior product to address that problem.
    • What the users think they want will change as soon as they see what we develop.
    -- Watts Humphrey, Some Programming Principles--Requirements, January 2003
    There is a very fancy technical term biologists use to describe completely stable systems. This highly sophisticated technical term is the word “DEAD!” Change is not the enemy – stagnation is!
    -- from The Unchangeable Rules of Software Change
    Requirements changes late in the lifecycle are competitive advantage, IF you can act on them!
    -- Mary Poppendieck
    Becoming agile means accepting uncertainty about the future as a way of dealing with the future. Any project development effort should be a balance between anticipation (planning based on what we know now) and adaptation (responding to what we learn over time).
    -- James Highsmith, Embrace Uncertainty
    Scrum’s Uncertainty Principle is: Customers don’t know what they want until they see it, and they always reserve the right to change their mind.
    -- Jeff Sutherland, Emergence of Essential Systems
    Systems requirements cannot ever be stated fully in advance, not even in principle, because the user doesn’t know them in advance--not even in principle.

    To assert otherwise is to ignore the fact that the development process itself changes the user’s perceptions of what is possible, increases his or her insights into the applications environment, and indeed often changes that environment itself.

    We suggest an analogy with the Heisenberg Uncertainty Principle: any system development activity inevitably changes the environment out of which the need for the system arose. System development methodology must take into account that the user, and his or her needs and environment, change during the process.
    -- Dan McCracken and Michael A. Jackson, ACM SIGSoft, 1982
Is your favorite quote about software change/uncertainty missing from the above? Leave a comment and tell me about it!
Posted by at No comments:
Labels:

Wednesday, June 24, 2009

Technical Debt - Definition and Resources

I ran across a few really good papers on the subject of technical debt that are fairly comprehensive in their treatment of not just what it is, but also how to manage it:
For those not already well-versed on the subject, Technical Debt [a.k.a. Design Debt] occurs when our software becomes difficult or risky to change, and takes increasingly more time & effort to evolve. Technical debt represents the cost of the accumulated amount of rework that will be necessary to correct and/or recover from the deviation between:
  • the current design of the system, versus ...
  • a design that is minimally complex yet sufficiently complete to ensure correctness & consistency for timely delivery.
This effort grows more than linearly over time as a system becomes bigger and more complex.

The economic impact of technical debt is directly related to the cost of complexity and its resulting “friction” against the velocity of the development team (and, subsequently, upon the ease of system evolution).

Technical debt can be caused by under-engineering just as much as it can be caused by overengineering (overdesigning). It is a difficult, delicate and dynamic balancing act to achieve the necessary and sufficient amount of design to implement only the essential complexity required by system:
  • Sometimes we knowingly (under great pressure) do something the "quick & dirty" way, with the intent to "do it right" later (but not too late).
  • Sometimes we try to do too much to soon, and elaborate or implement details of the requirements when we and our customers/users haven't yet learned enough about the true needs of the system.
  • Sometimes we haven't yet learned enough about good design, and unintentionally violate design principles, resulting in undesirable dependencies that make code or other work-products hard to change.
  • Sometimes we neglect to properly "tend" to the design and don't give it the necessary amount of ongoing care and feeding needed to keep it "fit" and "supple."
But whether it is a deviation between principles and practice (knowingly or unknowingly), guessing incorrectly, anticipating too early, neglect, poor quality, or even just the laws of software evolution, we must make plans and take action to deal with the inevitable entropy of evolution or else we will sink too far into technical debt.

That's just my two cents on the subject of course. What do others have to say about it?

Wikipedia defines Technical Debt by referring to the words of Ward Cunningham, who first drew the comparison between technical complexity and debt in a 1992 experience report:
"Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite.... The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise."

On the C2 Wiki, Ward defined Technical Debt as:
“Technical Debt includes those internal things that you choose not to do now, but which will impede future development if left undone. This includes deferred refactoring. Technical Debt doesn't include deferred functionality, except possibly in edge cases where delivered functionality is "good enough" for the customer, but doesn't satisfy some standard (e.g., a UI element that isn't fully compliant with some UI standard).”

Martin Fowler's definition of Technical Debt is also frequently cited:
"Doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future."

In his introduction to Refactoring to Patterns, Joshua Kerievsky quite simply defines design debt as follows:
Design debt occurs when you don't consistently do three things.
  1. Remove duplication.
  2. Simplify your code.
  3. Clarify you code's intent.
Few systems remain completely free of design debt. Wired as we are, humans just don't write perfect code the first time around. We naturally accumulate design debt ....

Due to ignorance or a commitment to "not fix what ain't broken," many programmers and teams spend little time paying down design debt.... In financial terms, if you don't pay off a debt, you incur late fees. If you don't pay your late fees, you incur higher late fees. The more you don't pay, the worse your fees and payments become. Compound interest kicks in, and as time goes on, getting out of debt becomes an impossible dream. So it is with design debt.

James Shore describes Design Debt in much the same manner:
When the cost of change increases, it's because the design quality is decreasing. Since retiring or rewriting software means writing off a huge investment, you wouldn't expect any team to let design quality deteriorate to that point. Yet it happens all the time. Why?

Design debt" explains the problem. When a team is working under pressure, they take shortcuts that compromise design quality. It's like taking out a high-interest loan. The team gets a short-term boost in speed, but from that point forward, changes are more expensive: they're paying interest on the loan. The only way to stop paying interest is to pay back the loan's principle and fix the design shortcuts.

Steve McConnell defines Technical Debt as follows:
“Technical Debt” refers to delayed technical work that is incurred when technical short cuts are taken, usually in pursuit of calendar-driven software schedules. Just like financial debt, some technical debts can serve valuable business purposes. Other technical debts are simply counterproductive. The ability to take on debt safely, track their debt, manage their debt, and pay down their debt varies among different organizations. Explicit decision making before taking on debt and more explicit tracking of debt are advised.

Mary Poppendieck gives a definition of Technical Debt in her upcoming book Leading Lean Software Development:
“All successful software gets changed. So if we think we’re working on code that will be successful … we need to keep it easy to change. Anything that makes code difficult to change is technical debt. Just like any other debt, the cost of paying off technical debt gets more and more expensive over time. … Technical debt drives the total cost of software ownership relentlessly higher … eventually we will have to pay it off or the system will go bankrupt.”

Kane Mar, in Technical Debt and Design Death, describes technical debt and later likens it to the effects of entropy:
Technical debt is simply defined as deferred work that is not directly related to new functionality, but necessary for the overall quality of the system. Examples of this include delaying upgrades to necessary tools and frameworks, delaying the refactoring of overly complex system components, and so on. If this technical work is necessary for the health of the system, then what happens when it is ignored for too long? What happens when several layers of cruft accumulate and there are no unit tests to aid refactoring? ...

In thermodynamics, “entropy” refers to the randomness of the components of a system. When the term is applied to software it is considered a measure of disorder. Thus entropy in software is the result of changes made to the code base, including bug fixes, updates to existing functionality, and the addition of new functionality. But over a period of time, these small changes can snowball to create a system that is difficult to change, overly connected to external systems, and lacks clear delineation of functionality.

Chris Sterling, in his upcoming book on Architecture in an Agile Organization, distinguishes between Software Debt and Technical Debt:
Software debt accumulates when focus remains on immediate completion while neglecting changeability of the system. The accumulation of debt does not impact software delivery immediately. At first it creates a sense of increased feature delivery with management, business stakeholders and the team. Business stakeholders respond well to the pace of delivered functionality. What they don’t understand is that this only represents an illusion of earlier returns on their investment.
...
This allows both the business and software delivery teams to live in the illusion of status quo far longer than they should. At some point previously small forms of decay in the system become large enough to affect our software delivery to the point that working harder and longer doesn’t result in success.
...
Debt is made glaringly visible when the team works on stabilizing the software functionality late in the release cycle. Integration, testing, and bug fixing is unpredictable and does not get resolved adequately before the release. People involved in the project stay late working to get the release out the door. It is now too late to pay back the debt accrued during the feature development.

The following sources constitute what I call software debt:
  • Technical Debt: those activities that a team or team members chooses not to do now and will impede future development if left undone
  • Quality Debt: diminishing ability to verify functional and technical quality of entire system
  • Configuration Management Debt: integration and release management become more risky, complex, and error-prone
  • Design Debt: cost of adding average sized features is increasing to more than writing from scratch
  • Platform Experience Debt: availability and cost of people to work on system features are becoming limited

Over at the Agile-in-a-Flash blog, Jeff Langr and Tim Ottinger provide a flashcard of tips & truths about technical debt, along with a more detailed discussion about its nature and origins. They distill several nuggets of wisdom like:
  • Incurring technical debt means your velocity slows down and you will deliver less value
  • The cost of getting out-of-debt is compounded over time: the longer you wait, the faster it grow!
  • If you plan to incur technical debt, the persons responsible must have a workable plan to pay it off!
  • “Interest only” payments won’t improve things
  • Pay early, pay often, and pay-as-you-go. (The only other options are bankruptcy or death.)
  • Remember: Those with the worst debt problems often have the most difficulty imagining a life without borrowing!

So how might we properly measure/monitor and account for technical debt? Well, some of the above papers I mentioned have some good ideas. Some that address it more explicitly are:
Here are a number of other very good resources on technical debt culled from blogs, articles, and presentations over the years. Some of them describe it, while others delve into methods for managing it and paying it off:
Posted by at 3 comments:
Labels: , ,

Sunday, June 21, 2009

Gee Scrum Method Owners!

[NOTE: I originally wrote this before Agile2009 (in Chicago), but forgot to press "publish", so it stayed a 'Draft' until I discovered this and published it much later]

 Sung to the tune of Gee, Officer Krupke! from West-Side Story -- Music: Leonard Bernstein, Lyrics: Stephen Sondheim (see video of the movie performance)

 Background:
For about a year now, I believe there has been a greater than usual amount of Agile-related blogging and mail-list discussion on the subject of the shortcomings of what is currently the most popular agile development method: Scrum. Some of them are legitimately about Scrum, others are really more about the kinds of problems one encounters introducing Agile methods without appropriate emphasis/execution of the technical practices (especially Refactoring and TDD).

It seems much of it may have started around the same time that Alan Shalloway (and later others, including Ron Jeffries, below) were banned from the ScrumDevelopment list for “going meta” and discussing problems with and/or additions to Scrum itself instead of how to apply/use it. Alan subsequently created the Lean-Agile list for such discussions (and is also working on a book).
As of this writing, more recent examples include the following:

Cast of Characters: You will need individual actors portraying the following roles!
NOTE that the people are real, but the dialogue is fictional, even tho it is derived from bits and pieces of actual messages and blog-posts.

Lyrics:
    RON (spoken): Hey, you!

    ALAN (spoken): Me, Officer Jeffries?

    RON (spoken):
    That’s Scrum Master Jeffries to you! Gimme one good reason for not draggin’ ya to the scrumdevelopment list to answer for blamin all yer troubles on Scrum. What’ll you say to Schwaber, ya punk!

    ALAN (sings):
    Dear kindly Scrum-Lord Schwaber, You gotta understand,
    It's just our agile nature that gets us Scrum-list banned.
    Our management is clueless, our backlog’s full of junk,
    Golly Moses, that’s where Scrum has flunked!

    ALAN with BRAD & PETER:
    Gee Scrum-method owners, we're very upset;
    Your method doesn’t scale and we’re in technical debt!
    We ain't no extremists, Our motives are clean,
    Deep down inside us there is Lean!

    ALAN: There is Lean!

    ALL:
    There is Lean, there is Lean, There is untapped Lean,
    Deep inside XP and Scrum there’s Lean.

    RON (spoken -- imitating Scrum list-owner):
    That's an off-topic discussion!

    ALAN (spoken):
    Lemme tell it to the world!

    RON (“Scrum list-owner”):
    Just tell it some place else!

    ALAN (singing, while creating the Lean-Agile list):
    Dear Lean-Agile list-readers, The Scrum-list is too closed.
    Despite their Agile leaders, the truth must be exposed:
    They didn't want discussion of what Scrum oughta add.
    Leapin' lizards, that's why I'm so “rad!”

    ALAN:
    Hey Scrum-method owners, you're really deep-fried!
    There’s plenty wrong with Scrum that needs Lean-thinking applied!

    PETER    : Those darn product-owners have gotta be curbed!
    RON: They’re pathologic'ly disturbed!
    BRAD: They’re disturbed?

    ALL:
    They're disturbed. They're disturbed, They're the most disturbed,
    Like they're pathologic'ly disturbed.

    ALAN (spoken, imitating Mary Poppendieck):
    Hear ye! Hear ye! In the opinion of Lean-thinking, these POs act depraved on account of they ain't had a proper “frame.”

    BRAD (spoken): Hey, they’re depraved on account of their dis-functional context!
    RON (spoken): Lemme to introduce ‘em to my fully functional foot!

    BRAD (singing, to Ron)
    My PO is so dastard, da coach won’t hear my plea,
    da team’s code ain’t refactored, ‘cuz PO won’t agree.
    They add it to the backlog for later to address,
    Goodness gracious, dat's why it’s a mess!

    RON:
    Please! “Bad product-owners!” that reason’s so lame!
    You shouldn’t even ask them, (and you should be ashamed)
    It’s really yer context that oughta be changed,
    It’s economic’ly deranged!

    BRAD: We’re deranged?

    ALL:
    We’re deranged! We’re deranged! We are so de-ranged,
    We're so economic’ly deranged!

    RON (spoken):
    In my opinion, Scrum don't need to have its head roles re-thunk at all. Use of RASCI matrices is surely a management disease!

    BRAD (spoken): Hey, I got a management disease!

    ALL (spoken sarcastically): So go buy an agile management tool.

    BRAD (singing to "Agile tool-vendor"):
    Dear kindly Scrum-tool master, they say go use your tools.
    They’ll save us from disaster, ‘cuz they’ll enforce da rules.
    And for our retrospectives, they’ll generate nice charts
    Glory Osky, that's why our code smarts!

    UNCLE BOB (with RON):
    Yikes! Stupid tool vendors, you've done it again.
    This team don't need a tool, just index cards and a pen.
    It ain't just enough to inspect-and-adapt;

    UNCLE BOB    : Value craftsmanship over crap!
    BRAD: Ship 'er crap?

    ALL:
    Don’t ship crap! Don’t make crap! Craftsmanship over crap!
    Craft clean code as if you give a crap.

    UNCLE BOB (spoken):
    We hereby declare this project’s code to be crap on account of its ignerantive and excremental development. Failure to write clean code is professionally irresponsible! TDD and refactoring are not optional!

    RON: The problem is your context!
    ALAN: The answer is go Lean!
    BRAD: The problem is design debt!
    BOB: The answer is code clean!
    PETER: The problem is Scrum's growing!
    ALAN: The answer is Scrum's grown!

    ALL:
    Scrum Lords we got troubles of our own!
    To all unclean-coders We just have to yell,
    “Your projects can’t be agile if your code-bases smell!”
    Gee, Scrum method-owners, What are we to do!
    Dear Scrum method-owners, SCRUM YOU!!

Acknowledgements:

I credit Ron Jeffries with my inspiration (obsession?) in crafting the above! Ron quoted a line from the West-Side Story song “Gee Officer Krupke!” in one of his email replies and I couldn’t get the song out of my head all that day and began piecing together the above (which I developed iteratively & incrementally using TDD and refactoring – of course :-).
Posted by at No comments:
Labels: , , ,

Saturday, June 20, 2009

Resources on Self-Organizing Teams for Agility

In the past several blog-entries I've been focusing on the agile principle of self-organization, what it means, and what it implies for teams. So far, I've written about Agile Self-Organization versus Lean Leadership, Self-Organization and Complexity, and Agile Self-Organizing Teams.

For some online resources (articles and presentations) on self-organization and self-organizing teams, I recommend the following:

I'll be taking a break on this topic (Self-Organization) for awhile, and then returning to it again later to talk about Swarm Behavior, Collective Intelligence, Social Creativity, Group Coherence, Group Decision-Making and Holocracy/Sociocracy.
Posted by at No comments:
Labels: , ,

Tuesday, June 16, 2009

Agile Self-Organizing Teams

The previous blog-entry on self-organization was lots of jargon and technical mumbo jumbo that didn't say too much about what that means for teams of people. So let's shift from talking about self-organizing systems in complexity science to talking about how it applies to self-organizing teams in an agile context.

A self-organizing team is a team that is led and organized by it's members, to attain goals and objectives specified by management within the constraints of its environment:
  • Management can shape and "nudge" the team and its members, but management doesn't try to dictate the details of "what" the solution is nor the process of how to create it.
  • The team is responsible for not only leading and organizing itself to achieve its goals, but also to monitor and adapt its behavior to correct/improve its own performance.
  • This means the team can change how it leads and organizes itself in order to respond to feedback and constraints from its environment, which also implies that ...
  • There is no single central "leader" for the team over the lifetime of the team/project - the "leader" is not a static assignment, but rather a dynamic role
  • So the person(s) leading any given moment may change, depending on the particular decision, activity, or problem being addressed in any particular context/situation.

By themselves, self-organizing teams are neither "good" nor "bad." They simply "are." They require a supporting management environment (the "fitness landscape") and organizational culture that establishes, communicates, rewards and reinforces the "right" set of values and principles. Without supportive management and the proper leadership culture, there is a very high likelihood that a self-organizing team may be unable to create good results or effective processes (or both). In fact, it's not uncommon for a newly formed & "empowered" self-organizing team to fall into many of the same dysfunctional patterns of behavior that it was most trying to escape from within the "management" that only recently "empowered" the team.

An "agile team" is (supposed to be) a self-organizing team that is guided by the agile values and agile principles (given by the agile manifesto) and is supported by a trusting and empowering style of management. With management supporting their agile values/principles, Agile teams "self-organize" to collectively decide and do what is needed in order to: make and meet commitments, develop a quality product, respond to feedback, and adapt to changes.

So an Agile Self-Organizing Team is:
  • Autonomous: There is no single central decision-making authority. Control is distributed collectively to the team.

  • Adaptive: The team dynamically adjusts as needed across roles, functional specialties, and other boundaries, in order to solve their own problems and improve their own performance.

  • Accountable: The team collectively shares responsibility for results, and members hold each other accountable for outcomes.
Here are some choice quotes regarding self-organizing teams ...

“The team makes most decisions, while every member could step in and become leader in specific areas and situations. People are highly capable, committed and self-driven.”
—Andriy Solovey, What is the best leadership style for the software team?

“This causes a shift in the roles of managers from planning, controlling, directing, and managing to new roles like building trust, facilitating and supporting team decisions, expanding team capabilities, anticipating and influencing change.”
—Diana Larsen, Exploring Self-Organizing Software Development Teams

"Responsibility-Based Planning and Control: Respecting people means that teams are given general plans and reasonable goals and are trusted to self-organize to meet the goals. Respect means that instead of telling people what to do and how to do it, you develop a reflexive organization where people use their heads and figure this out for themselves."
—Mary Poppendieck, Implementing Lean Software Development

In Learning is the Bottleneck, Amr Elssamadisy & Deborah Hartmann write:
"Human psychology aspect adds that self-organized teams:
  • are more responsible for end results, self-disciplined and self-driven
  • avoid dependency on the formal leader qualities
  • motivated, initiative and willing to act
  • enjoy work more
  • better insured against groupthink, conformity and diffusion of responsibility
  • not shifting judgment and decisions to others, better in finding alternative and balancing options
  • every member is in charge, ready to step in as a leader and have incentive to develop leadership skills
A self-organized team is possible when people carry shared purpose, principles and values. They support and respect each other. And they want to succeed. The [Agile] team works together to respond to changes that happen together. They collectively do what needs to be done to build the software."

In his 2001 paper Agile Processes and Self-Organization Ken Schwaber wrote:
"Agile processes employ self-organizing teams to handle the complexity inherent in systems development projects. A team of individuals is formed. They organize themselves into a team in response to the pressure of a deadline, reminding me of the saying, "Nothing focuses the mind like a noose!" The pressure cooker of the deadline produces cooperation and creativity that otherwise is rare. This may seem inhumane, but compared with non-agile practices for dealing with complexity, self-organization is a breath of fresh air."

This is what Kevin Kelly wrote about that problem in his book Out of Control:
"When everything is connected to everything in a distributed network, everything happens at once. When everything happens at once, wide and fast moving problems simply route around any central authority. Therefore overall governance must arise from the most humble interdependent acts done locally in parallel, and not from a central command."

Roger Lewin wrote in Complexity: Life at the Edge of Chaos:
"Complexity science implies that CEOs and managers must give up control -- or rather, the illusion of control -- when they are trying to lead their organization to some goal. But they do need to create the environment in which creativity can emerge. The message of complexity science is not simply to stand back and wait for the right solutions to emerge. Too little control is just as misguided a business strategy as too much. Some structure is necessary. The degree and nature of control that CEOs establish in their companies strongly influences what emerges, in terms of culture, creativity, and adaptability."

Mishkin Berteig writes in Team Self-Organization:
"In agile teams, this concept of self-organization is taken quite far. Team members collaborate to get work done. No one orders a team or an individual to do specific work. The team members volunteer for work that they see needs doing, even if it is not something that is in their area of expertise. An agile team is constantly promoting learning in its people. Agile teams are also cross-functional so that the team can get work done without relying on external services. The team therefore represents a complete work unit capable of taking a function valuable to customers from start to finish, from idea to deployment."

In Why Agile Principles are Important, Simon Baker writes:
An experienced agile software development team is a highly social group that is self-organising around these principles and acts with coordination and collective behaviour. This collective behaviour comprises:

  • Collective mind where individual team members develop shared understandings of the team's tasks and of one another, and come to understand how their work contributes to the work of the team thereby facilitating team performance.

  • Swarm intelligence which gives a team the ability to adapt to changes, and robustness which enables them to still perform and deliver even when one or more members fail.

From Of ants and men: self-organized teams in human and insect organizations
To cope with today’s complex, fast-paced, and ever-changing business environment, companies need to shift their overall structure to produce adaptive, highly responsive organizations. The use of teams, particularly self-organized teams with their reactive, emergent properties, may be one way of achieving this goal.

In other words, insect societies often harness the power of self-organization such that with the appropriate set of feedbacks, interindividual interactions, and proximate mechanisms, group-level adaptive behavior simply emerges. No one directs the foragers where to find food, the network of trails and interactions takes care of that; individuals are not allocated to tasks, the reverse is true: the tasks allocate the workers.

McMillan-Parsons (1999) found that the teams fitted Stacey’s (1996) description of self-organizing groups or teams as ones that arise spontaneously around specific issues, communicate and cooperate about these issues, reach a consensus, and make a committed response to these issues. Further, ‘research suggested that self organizing teams have a strong sense of shared purpose, strong personal commitment, display creative and spontaneous behaviors, have high levels of energy and enthusiasm, and that an inherent order emerges from their activities’.

Importantly, in self-organizing teams the members self select and there is no-one checking to see if they have the necessary range of attributes. In her study, McMillan (1999) discovered that members of the self-organizing teams studied learned new skills and developed new attributes to meet the needs of the team.

One characteristic of such organizations is adhocracy. These large, mature yet high-performing companies manage to generate the flexible and adaptive properties of smaller entrepreneurial organizations—in short, to “be big and yet to act small at the same time”. Using teams is one key means of achieving that, for, as Flory (2002, p. 9) remarks, self-managed teams, ‘are fast moving, fast learning groups, flexible, highly autonomous and have a well-developed pro-active attitude and sense of responsibility. These characteristics are the very reason they are brought into life as answers for organizations to respond to a fast moving world.’


Self-managed Teams Self-organized Teams
Part of formal organization structure Not part of formal organization structure
Formal, temporary, or permanent Informal and temporary
Not spontaneously formed Formed spontaneously around issue(s)
Indirectly controlled by senior management Boundaries influenced by senior management
Managers decide ‘who’ and ‘what’ Team members decide ‘who’ and ‘what’
Replace the hierarchy Often in conflict with or constrained by the hierarchy
Empowered by senior management Empowered by the team’s members
Strongly shared culture Cultural differences provoke and constrain
Some sense of shared purpose Strong sense of shared purpose
Order created via recognized processes Inherent order emerges
Behaviors influenced by procedures and roles Spontaneous, creative behaviors
Strong sense of team commitment Strong sense of personal commitment
Some energy and enthusiasm High levels of energy and enthusiasm
Decision making is mainly a planned process Decision making is mainly a spontaneous process
At least one member’s primary role is organizational All members’ primary role relate to the task


In my next blog-entry I'll give links to several other resources on Self-Organizing Teams.
Posted by at 1 comment:
Labels: ,

Sunday, June 14, 2009

Self-Organization and Complexity

In my previous blog-entry I talked a little about how self-organization is a key aspect of software agility. In this blog-entry I'd like to explore in more detail just what "self-organization" really means.

Self-organization comes from complexity science and the theory of complex adaptive systems (CAS). A system is "self-organizing" if, left to its own devices, it tends to become more organized over time. This is in stark contrast to the concept of entropy from the laws of thermodynamics whereby a closed system tends to move to a state of increasing disorder in the absence of outside influences.

In a complex adaptive system where self-organization occurs, we necessarily have an open system rather than a closed one. The theory goes that if a complex system possesses the necessary emergent properties in an appropriate fitness landscape, then the system will yield emergent behavior, exhibiting system-wide "patterns", that increases the "order" or organization in the system. Hence the system has become self-organizing as a result of this emergent behavior & structure.

Some of the emergent properties of self-organizing systems can include:
  • Autonomy,
  • Adaptability,
  • Decentralization,
  • Dynamic reconfiguration,
  • Positive & Negative Feedback
  • Cooperation & Information exchange
The theory of complex systems suggests that self-organized systems are:
  • better in selecting optimal state and local decisions
  • robust
  • effectively adapt to environment and use feedback loops
  • often come up with emergent and unexpected solutions
  • self-regulated and better cope with perturbations
According to James Highsmith,
“There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback”

The Emergent Phenomena Research Group at Brywn Mawr says:
"In self-organizing systems, global order (i.e., complex aggregate structure/behavior) results (emerges) from the local behaviors of simple agents following simple rules specifying conditions under which those agents act (interact) in such a way that the results of the agents' actions are fed back as input to the operation of the rule system at some future time step. "

In The Science of Self-organization and Adaptivity, Francis Heylighen writes:
Self-organization is basically the spontaneous creation of a globally coherent pattern out of the local interactions between initially independent components. This collective order is organized in function of its own maintenance, and thus tends to resist perturbations. This robustness is achieved by distributed, redundant control so that damage can be restored by the remaining, undamaged sections. ...

Self-organizing systems are characterized by: distributed control (absence of centralized control), continual adaptation to a changing environment, emergent structure from local interaction, robustness/resiliency (able under change and can quickly repair, correct, adapt/adjust), non-linearity, feedback (both positive and negative), self-sufficiency and closure/coherence. ...

Organizational closure turns a collection of interacting elements into an individual, coherent whole. This whole has properties that arise out of its organization, and that cannot be reduced to the properties of its elements. Such properties are called emergent.

Every self-organizing system adapts to its environment; Systems may be called adaptive if they can adjust to such changes while keeping their organization as much as possible intact.

From Software Development is Complex Adaptive System. No Doubt
Emergence is the way complex systems and patterns arise out of a multiplicity of relatively simple interactions. Small actions of agents lead to unexpected emergent system behavior and it is impossible to predict system behavior based on the behavior of an individual agent. Small errors pile up and may cause huge problem.

There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback.

A system in equilibrium does not have the internal dynamics that enables it to respond to the environment and will slowly (or quickly) die. A system in chaos stops functioning as a system. The most productive state to be in is at the edge of chaos where there is maximum variety and creativity, leading to new possibilities. A set of simple rules allows edge of chaos and powers creativity, flexibility and success.

And last but not least, from Conditions for Self-Organizing in Human Systems :
Self-organization is the spontaneous generation of order in a complex adaptive system. It is the process by which the internal dynamics of a system generate system-wide patterns. Some of the emergent patterns in a self-organizing system are coherent, and others are not. Coherence is a state of the system in which:
  • Meaning is shared among agents.
  • Internal tension is reduced.
  • Actions of agents and sub-systems are aligned with system-wide intentionality.
  • Patterns are repeated across scales and in different parts of the system.
  • A minimum amount of energy of the system is dissipated through internal interactions.
  • Parts of the system function in complementary ways.
System-wide patterns in which the parts are aligned and mutually reinforcing (coherent) are more stable than other self-organized patterns. Because of the mutually reinforcing dynamics of a coherent pattern, the effort required to change the pattern is greater than the effort to maintain it, so coherent patterns are more stable than incoherent ones. When the system reaches a state of coherence ... tensions within the system are reduced, and the available energy of the system is aligned and focused on system-wide behaviors, rather than diverse and disruptive behavior of individual agents or sub-system clusters. ...

In human systems, the process of self-organizing is particularly important. Teams, institutions, and communities include individuals or groups of individuals that function as agents in self-organizing. As the agents interact, patterns of behavior emerge over time.These patterns form and reform spontaneously and continually at multiple levels within the system. Individuals work together to form teams. Ethnic identity groups establish relationships and micro-cultures. Functional departments engage with each other to do the work of the organization. At all of these levels, agents interact naturally to form patterns of system-wide behavior. ...

The CDE model is a set of the three conditions for self-organizing of human systems: Container, significant Difference, and transforming Exchange. The path, rate, and outcomes of self-organizing processes are influenced by these three conditions, which are co-dependent such that the function of each of the conditions depends on the others in nonlinear interactions in the system. A change in any one of the conditions results in a change in the other two over time.

Also see the following:
In my next blog-entry I'll talk specifically about self-organizing teams. Some specific characteristics and/or results of self-organization that I'll be delving into more deeply in subsequent blog-entries are:
  • Swarm Behavior (a.k.a. Swarm Intelligence)
  • Collective Intelligence (a.k.a. Collective Mind)
  • Social Creativity
  • Group Coherence
Posted by at 1 comment:
Labels: ,

Friday, June 12, 2009

Agile Self-Organization versus Lean Leadership

Getting back to the agility cycle ... recall that I started with the business agility cycle and used that to derive the software agility cycle. There isn't a great deal of difference between the first two steps of the business-agility cycle and the software-agility cycle, other than the fact that much of the former takes place at a higher-level of organizational management and strategy.

The biggest difference between the two cycles happens after those first two steps. In the business agility cycle we then decide-communicate-act, suggesting that it is the higher-ups who make and communicate the decisions and the lower-end of the organizational food chain (the "worker bees") that execute the organization's strategic solution.

If that seems a bit command-and-control, its because it is (a bit). It's also likely necessary in larger organizations where it is virtually impossible to avoid having 2 or more layers of management. There, the "communicate & act" steps are really the process of providing focus, and communicating objectives and constraints to the knowledge workers, and letting them apply principles of collaboration and self-organization to solve the problem (like in the software-agility cycle).

So the key difference between business-agility and software-agility is the extra emphasis of the latter on "the people factor" and on the notion of dynamic self-organization of knowledge-workers as empowered, self-organizing teams. This difference between agility at the business-level versus software-level is also the key difference between Lean-vs-Agile:
  1. Lean focuses more on flow whereas Agility focuses more on adapting to change (this is true of both software agility and business-agility). As it turns out, focusing on flow requires being able to adapt to change; and focusing on adaptiveness and responsiveness requires a focus on flow. So here, the end-results may be quite similar

  2. Agile software development emphasizes a very hands-off management style of not just trusting and empowering the team(s), but often to the extreme of basically saying "leave us alone and don't interfere" to management (and in return promising extreme transparency and frequent tangible end-results).

  3. Much of this is due to the fact that the scope of Agile development is limited to software projects and teams, whereas Lean is targeted at enterprise-wide scale across the whole value-stream.

That is not to say that Lean (or business-agility) don't emphasize trusting and empowering workers and teams (they do). But they don't have the underlying inherent attitude of "just trust us and stay out of the way." The "trust" often doesn't seem to extend as far in the other direction with Agile development (e.g. not trusting management direction to the same extent that management is asked to trust the team).

I think this stems from the fact that software agility started more at the grass-roots level, with developers and teams struggling to create good, quality software in the face of what was all too often a fundamentally broken management framework for the governance, visibility, and lifecycle of software development projects. Because they were working and trying to get support from the bottom-up, they needed to be shielded from and unshackled by all the dilbert-esque "pointy haired managers" and their big, bad governance framework with its "evil" waterfall lifecycle.

And in that context, the advice of "just get out of the way and trust us and we promise to deliver working software every 2-4 weeks" is actually a very effective strategy to employ. When management doesn't "get it", their attempts to steer, direct and intervene are often the equivalent of "friendly fire" (which, despite well intentions, still yields calamitous results.)

But Lean comes from a history of a much more enterprise-wide scope, often even using a top-down deployment strategy. So when it asks management to trust and empower workers and teams it expects the corresponding change in its leaders and their leadership-style, and for them to still play a strong, active and participative role with their projects and teams.

Agile teams often get a "bad rap" for having this isolationist attitude toward management. And sometimes that "rap" is warranted. But when the leadership "gets it" and understands and values the "new" paradigm of why+how agile works, and why+how the "old way" didn't, then it probably is time for the leadership to play a different, more active role while still trusting and empowering teams (and still enabling self-organization). This is the view that Lean takes toward management, and is a big part of why it is better received by management and is more broadly applicable for scaling agility "up" and "out" in larger enterprises.
Posted by at 8 comments:
Labels: , , ,
Subscribe to: Posts (Atom)