Sunday, June 14, 2009

Self-Organization and Complexity

In my previous blog-entry I talked a little about how self-organization is a key aspect of software agility. In this blog-entry I'd like to explore in more detail just what "self-organization" really means.

Self-organization comes from complexity science and the theory of complex adaptive systems (CAS). A system is "self-organizing" if, left to its own devices, it tends to become more organized over time. This is in stark contrast to the concept of entropy from the laws of thermodynamics whereby a closed system tends to move to a state of increasing disorder in the absence of outside influences.

In a complex adaptive system where self-organization occurs, we necessarily have an open system rather than a closed one. The theory goes that if a complex system possesses the necessary emergent properties in an appropriate fitness landscape, then the system will yield emergent behavior, exhibiting system-wide "patterns", that increases the "order" or organization in the system. Hence the system has become self-organizing as a result of this emergent behavior & structure.

Some of the emergent properties of self-organizing systems can include:
  • Autonomy,
  • Adaptability,
  • Decentralization,
  • Dynamic reconfiguration,
  • Positive & Negative Feedback
  • Cooperation & Information exchange
The theory of complex systems suggests that self-organized systems are:
  • better in selecting optimal state and local decisions
  • robust
  • effectively adapt to environment and use feedback loops
  • often come up with emergent and unexpected solutions
  • self-regulated and better cope with perturbations
According to James Highsmith,
“There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback”

The Emergent Phenomena Research Group at Brywn Mawr says:
"In self-organizing systems, global order (i.e., complex aggregate structure/behavior) results (emerges) from the local behaviors of simple agents following simple rules specifying conditions under which those agents act (interact) in such a way that the results of the agents' actions are fed back as input to the operation of the rule system at some future time step. "

In The Science of Self-organization and Adaptivity, Francis Heylighen writes:
Self-organization is basically the spontaneous creation of a globally coherent pattern out of the local interactions between initially independent components. This collective order is organized in function of its own maintenance, and thus tends to resist perturbations. This robustness is achieved by distributed, redundant control so that damage can be restored by the remaining, undamaged sections. ...

Self-organizing systems are characterized by: distributed control (absence of centralized control), continual adaptation to a changing environment, emergent structure from local interaction, robustness/resiliency (able under change and can quickly repair, correct, adapt/adjust), non-linearity, feedback (both positive and negative), self-sufficiency and closure/coherence. ...

Organizational closure turns a collection of interacting elements into an individual, coherent whole. This whole has properties that arise out of its organization, and that cannot be reduced to the properties of its elements. Such properties are called emergent.

Every self-organizing system adapts to its environment; Systems may be called adaptive if they can adjust to such changes while keeping their organization as much as possible intact.

From Software Development is Complex Adaptive System. No Doubt
Emergence is the way complex systems and patterns arise out of a multiplicity of relatively simple interactions. Small actions of agents lead to unexpected emergent system behavior and it is impossible to predict system behavior based on the behavior of an individual agent. Small errors pile up and may cause huge problem.

There's no hierarchy of command and control in a complex adaptive system. There's no planning or managing, but there's a constant re-organizing to find the best fit to the environment. The system is continually self-organizing through the process of emergence and feedback.

A system in equilibrium does not have the internal dynamics that enables it to respond to the environment and will slowly (or quickly) die. A system in chaos stops functioning as a system. The most productive state to be in is at the edge of chaos where there is maximum variety and creativity, leading to new possibilities. A set of simple rules allows edge of chaos and powers creativity, flexibility and success.

And last but not least, from Conditions for Self-Organizing in Human Systems :
Self-organization is the spontaneous generation of order in a complex adaptive system. It is the process by which the internal dynamics of a system generate system-wide patterns. Some of the emergent patterns in a self-organizing system are coherent, and others are not. Coherence is a state of the system in which:
  • Meaning is shared among agents.
  • Internal tension is reduced.
  • Actions of agents and sub-systems are aligned with system-wide intentionality.
  • Patterns are repeated across scales and in different parts of the system.
  • A minimum amount of energy of the system is dissipated through internal interactions.
  • Parts of the system function in complementary ways.
System-wide patterns in which the parts are aligned and mutually reinforcing (coherent) are more stable than other self-organized patterns. Because of the mutually reinforcing dynamics of a coherent pattern, the effort required to change the pattern is greater than the effort to maintain it, so coherent patterns are more stable than incoherent ones. When the system reaches a state of coherence ... tensions within the system are reduced, and the available energy of the system is aligned and focused on system-wide behaviors, rather than diverse and disruptive behavior of individual agents or sub-system clusters. ...

In human systems, the process of self-organizing is particularly important. Teams, institutions, and communities include individuals or groups of individuals that function as agents in self-organizing. As the agents interact, patterns of behavior emerge over time.These patterns form and reform spontaneously and continually at multiple levels within the system. Individuals work together to form teams. Ethnic identity groups establish relationships and micro-cultures. Functional departments engage with each other to do the work of the organization. At all of these levels, agents interact naturally to form patterns of system-wide behavior. ...

The CDE model is a set of the three conditions for self-organizing of human systems: Container, significant Difference, and transforming Exchange. The path, rate, and outcomes of self-organizing processes are influenced by these three conditions, which are co-dependent such that the function of each of the conditions depends on the others in nonlinear interactions in the system. A change in any one of the conditions results in a change in the other two over time.

Also see the following:
In my next blog-entry I'll talk specifically about self-organizing teams. Some specific characteristics and/or results of self-organization that I'll be delving into more deeply in subsequent blog-entries are:
  • Swarm Behavior (a.k.a. Swarm Intelligence)
  • Collective Intelligence (a.k.a. Collective Mind)
  • Social Creativity
  • Group Coherence

1 comment:

Wilfried Elmenreich said...

Self-organizing systems are sometimes literally (mis-)understood as some entity that organizes itself, like a person managing her own affairs in life. The "self" can be misleading here since it may be understood as a single controlling entity. See So when do we call a system self-organizing? for a follow-up discussion on this topic.