DesignPrinciples

Solar Digital, Inc.

Solar Digital was founded in 1998 to research and develop advanced techniques in software design and system architecture. While many companies focus on an exit strategy or build technology around a limited market segment, Solar Digital focused on a long-term strategy developing widely applicable intellectual property around sound engineering principles. Solar Digital developed its business and licensing model specifically to avoid the common pitfalls that plague commercial organizations and their respective technology. The end result is a unique software platform that has a strong value proposition across many market spaces. This document outlines the engineering design principles used in the development of Solar Digital’s technology.

Complexity and Unification

Complexity is a form of cost, and managing and reducing cost is a critical part of the engineering life cycle for software development. Flexibility is the ultimate design goal of many frameworks, however increased flexibility is inseparably related to increased complexity. Over the long term, the delicate balance between complexity and flexibility determines the success or failure of most software systems.

Many organizations extend the capabilities of their systems on a regular basis (specialization), but doing so introduces tension and complexity into the design of a system. If design tension is not managed, it becomes increasingly difficult to continue extending the value of a given system. A critical technique for managing complexity and design tension is Unification. Unification is an iterative process that performs relaxation on the design tension, helping to rationalize the architecture by raising the order of the system. An increase in order is the only effective way to achieve increased functionality while minimizing system complexity. Specializing a system by raising the order of its design is a form of strategic evolution, while specializing without Unification leads to convolution and a corresponding reduction in the degrees of freedom that a system can express.

Unification is a multi-step process:

1)Specialization - extend the capabilities of an existing system to solve new problems, increasing the expressive yield of the system
2)Identification - Identify common patterns and relationships that resulted in the development of these new capabilities
3)Generalization - Capture those patterns and relationships with unifying abstractions and improved interfaces
4)Consolidation - Re-factor the technology to embrace these more evolved constructs, and eliminate any redundant representations in the system
5)Stabilization - Apply quality assurance cycles to return the system back to a reliable state

This iterative process systematically removes unnecessary complexity from a system and results in the creation of highly organized and reusable components which isolate and encapsulate expert knowledge. Unification can be viewed as a continual investment in the infrastructure of a system to increase its expressive yield while avoiding the costs of naive exploitation. It is not possible to avoid the iterative nature of unification and achieve the same results, because the insight required to develop mature and capable abstractions stems from developing solutions that are initially naive. Unfortunately many commercial organizations are pressured to commit such naive solutions to actual field use. This creates the immediate pressure and obligation to maintain and extend those naive solutions in their current form. As a result, future development choices are often compromised which leads to a cycle that is hard to break. As it turns out, Unification is extremely inconvenient to perform in commercial settings for the following reasons:

1)Re-factoring a system invariably breaks it in many ways and often requires many additional quality assurance cycles to re-stabilize the technology
2)New instabilities put at risk a company’s capacity to release maintenance builds that are guaranteed to improve customer satisfaction with a current product. Many organizations don’t have the resources to re-factor systems while at the same time support stable commercial systems, as this can double the cost of product development
3)Software systems that allow additional specialization by the customer or the user base of a technology can create significant dependancies on the way the system currently operates in its naive form. In these situations, there can be immense pressure for the developer of the base technology not to significantly change the nature of how the system operates. This can eliminate the possibility for this base system to embrace new abstractions and improved technical infrastructure.
4)After all the work involved in re-factoring and re-stabilizing software, there is no immediate or obvious benefit to the business. Gains are strategic in nature and only prove their value indirectly over time, so businesses do not see any immediate dividend

For these reasons and others, most organizations inevitably circumvent the cost of unification and are lured into a tactical cycle of supporting and over-specializing their technically naive solutions. This continual exploitation of a technology to achieve short term gains has an unavoidable long term cost.

Its worth noting that the unification cycle is closely related to the well established concept of an OODA loop (Observe, Orient, Decide, Act), which is a well known model important to both business and military strategy. Both models suggest that the success of a system over time is inexorably tied to the rate at which actionable information is processed and factored into the formulation of action and strategy. We refer to this as the rate of convergence, where highly convergent systems iterate at a higher frequency and thus evolve at a greater rate. More importantly, such systems evolve to maximize long-term strategic outcomes as opposed to short term tactical gains.

Divergence

Many companies intend to allow for the re-factoring of their technology. However, they normally concede to deadlines, market pressure, outside dependancies, and the need for revenue at the expense of not following a disciplined long-term engineering strategy. This results in missing key opportunities to re-factor their system in important ways. Over time, redundancy accumulates and compounds. Artificial diversity spreads throughout the system and the cost of managing and extending the system continues to rise. Without Unification, complexity eventually diverges and becomes too significant to address. The system becomes riddled with artificial dependancies that make it highly fragile and self-referential. Addressing problems at any one level ends up having too many unintended consequences. Ultimately, re-factoring the system becomes an intractable problem.

Human resources play a direct role in determining the rate of divergence of a system. The following factors contribute to divergence and are difficult problems to overcome:

1)In many organizations, multiple developers contribute to the different components of a system. In most cases, its not feasible for any single developer to have a detailed understanding of every component’s implementation. This lack of coherence can keep major opportunities for re-factoring out of reach. This problem grows rapidly as the number of system engineers is increased.
2)Employee turnover within the engineering staff creates additional incoherence. New developers will often never have the same depth of understanding as those that originally authored a system. This puts new developers at a significant disadvantage when it comes to re-factoring an existing heritage system.
3)Engineers often actively undermine system design for different reasons, such as working outside their skill level or having a tendency to create unnecessarily complicated solutions. Its also common in organizations for engineers to compete with each other for control of the systems architecture. Brilliant engineers often revel in their own ability to manufacture and manage complexity, which can help them gain ‘ownership’ over different parts of a system while keeping others at a disadvantage. These kinds of behaviors can create an arms race where the intentional and artificial increase in complexity undermines the technology and eventually the organization itself.

Life Cycle vs Life Span

When unification is not applied regularly to the engineering of an inherently complex system, complexity will eventually reach a threshold where the cost of maintaining and extending the system exceeds its value to a business. Such technology falls victim to an unnecessary and artificially limited life span. If the business is flourishing, it may have the resources to perform a complete rewrite of its technology. In such cases, it may attempt to re-factor the entire system at once, or it may just replace the system altogether. Even when this is possible, it is not as effective as re-factoring the smaller parts of the system over time in an iterative fashion. Important opportunities to make subtle choices that slowly influence the nature of the technology have already been lost, circumventing an important evolutional feedback loop. A major rewrite also takes much longer than anyone anticipates, and in some cases can actually bring down a business due to cost and time overruns. However, most organizations never find a complete rewrite possible anyhow, and in such cases the life span of the technology can often determine the life span of the organization or the projects its involved in.

Evolutionary System Design

Evolutionary system design (ESD) is what we call the engineering discipline where Unification is an integral and regular part of the development process. The Genesis, Unity, and xMach frameworks are the result of applying this evolutionary engineering process over more than a decade. ESD promotes a constant interplay between specialization and generalization throughout each iteration of the development process. After a system has been subjected to significant iterations of ESD, the pressures of adaptation forge reliable abstractions that reflect natural principles that are pervasive to computer science, and in many ways start to mimic natural patterns found in real world systems. Replacing weaker abstractions with more powerful ones is a form of natural selection in the design space of a system. In complex dynamic systems, this design process quickly evolves into a form of ecological modeling. The abstractions discovered through this process are much more durable than would otherwise be possible because adaptation is a regular and integral part of the design cycle.

Second Order Strategies

Technology on the whole tends to evolve at an exponential rate, which is a concept well established and reflected by what is commonly known as Moore’s Law. To build software systems that can endure an environment where second order change is expected, systems must be engineered using second order design strategies. ESD is a second order design strategy because it encourages design patterns to emerge that accelerate the rate of adaptation itself, through the use of an iterative feedback loop. Accelerated adaptation is necessary for any system to sustain its value in an environment of second order change.

SOLAR DIGITAL is a registered trademark and service mark of Solar Digital, Inc.