Conceptual Architecture

Preface

Our approach to Conceptual Architecture in the Visual Architecting Process (VAP) has a long history, with roots in software architecture work we did at HP in the 1990's. Box and line diagrams were then and still are a widely practiced expression of "the software architecture." In the earliest incarnations of what became VAP, we added CRC descriptions (Cunningham and Beck) repurposing them to components or architectural elements to give a lightweight but sufficient characterization of the architectural elements. Which is to say, in what follows, you will see the familiar, though sometimes disparaged, "block diagram" and component-responsibility templates and may react with a sense of déjà vu. What we articulate here is, we believe, nonetheless worthwhile, not only in pulling into one place key ideas about Conceptual Architecture, but also because it provides unique insight into, and important positioning of, the too-often misunderstood and under-appreciated role of Conceptual Architecture in designing and evolving software intensive systems. These insights are applicable to architectures of systems and systems-of-systems more generally, but the presentation here is in terms of software architecture.

What is Conceptual Architecture?

"Conceptual Architecture" is the conceptual view(s) of the architecture of a system. It describes at a broad brushstrokes conceptual level the significant design ideas of the system. In particular, this view includes diagrams and text which identify, explicate, rationalize and contextualize the key structures and mechanisms of the system, and the relationships among them.

Conceptual Architecture expresses the key architectural elements and their relationships that give "shape" to the system. Each of these architectural elements, or abstractions, play a critical role in the system which is signified by their responsibility assignments. They collaborate and interact, and these relationships are a significant design consideration not only because interaction points (analogous to articulation points in physical systems) and "seams" in the system are non-trivial, but because interactions give rise to synergy that makes the system more than a simple aggregate of executing parts. The parts, their arrangement and concert, cohere within and give expression to the system "gestalt."   

Conceptual Architecture Diagram

The Conceptual Architecture Diagram renders the formative architectural abstractions (named boxes) and their interrelationships (lines). For complex systems, there may be a set of such diagrams, exploring the (de)composition of more complex, architecturally interesting architectural elements.

Component-Responsibility-Collaborator-Rationale (CRC-R) Descriptions

The Conceptual Architecture also identifies the responsibilities of each of the architectural elements. We advocate doing this on Component-Responsibility-Collaborator-Rationale (CRC-R) Descriptions* for each (conceptual architecture) component, so that the reasoning behind component design choices is captured along with the responsibilities (connecting the dots to desired outcomes, strategic decisions, lessons from experience and grounding assumptions, etc.), and dependencies and other relationships are also recorded. It is a simple and "just enough" format that matches the spirit and intent of Conceptual Architecture, supporting conception, communication and evolution of the key design ideas, including the fundamental organization, of the system.

Mechanism Sketches

Beyond the (set of) Conceptual Architecture Diagram(s) and CRC-R descriptions, architectural mechanisms (i.e., mechanisms of architectural significance) are sketched (in diagrams and descriptions) to conceive and convey their essential design nature -- the design intent, contextual assumptions, structure, and "how it works" dynamic considerations. The capabilities of the system emerge from the inter- and intra-working of the parts of the system, and mechanisms allow us to focus on specific processes within the system, conceptually (at this point) designing architecturally significant functions of parts of the system.

Architecturally significant mechanisms are those that have more diffuse or systemic impact or are make-or-break important to system outcomes. Many of the patterns in our field's literature formulate tried-and-true mechanism designs addressing very specific system capabilities (often internally focused at system sustaining and structural integrity concerns because these are common across systems irrespective of their specific user-facing functionality). A system has functions just like a body has functions. And many of these are internal system sustaining functions that have to do with continuity rather than serving any immediate external demand being made of the system.       

Together, the expressions of the Conceptual Architecture form a conceptual framework within which we conceive of, reason about, communicate and share, extend and reify and evolve the key design ideas of the system.  

Why: Motivation for the Conceptual Architecture View

Conceptual Architecture is a key medium for describing the "big picture" and essential design ideas of our system, helping others to more rapidly comprehend a complex system, how it relates to its context and how it is composed, and its critical mechanisms or interworking to achieve some key internal system capability essential to sustaining itself. The Conceptual Architecture Diagram serves as a high level map of our system, providing for navigation around complex systems, location of responsibilities, and identification of dependencies.

But Conceptual Architecture is also a focus of essential design work! As we conceive of the system structure, we're tackling questions of decomposition and modularity, with the dual of composition and emergent properties. We do so as to make the system organizationally and cognitively tractable, bringing order to the system and addressing system challenges. We might think of this as a kind of "social order" if we see its elements as agents of system responsibilities, working with cohorts to deliver system capabilities and in this interaction effecting emergent properties. But even if we don't go that far, we are at least bringing a kind of mechanical order, complete with hierarchically composed sub-orders, to the system.

Modular systems enable us to partition realms of uncertainty and experiment, separating them from more stable and understood areas of the system; to separate areas of the system that change for the same reason, from other areas of the system that have different change impulses; to create areas of focus for investments in highly tuned performance or extreme robustness and resilience; to separate elements for reuse or leverage; and to create units of work assignment. (Whether we organize development primarily by features or by components, it is helpful to have clear component ownership so that the structural health of the component is an explicit charter.)

Early on, we're dealing with our conceits of formative system elements -- on paper. We're crafting lo-fidelity sketch-prototypes of the system structure, and iterating across the conception of the system and our discovery and exploration of how we might conceive of, arrange and build the system. Because it is cheap to do, relying more on how generative our imagination is leveraging the grist of experience, patterns, capabilities in other systems and frameworks, and so forth, we can explore various alternatives, discovering elements and relationships, illuminating the system conception or possibilities we might offer in terms of system capabilities. We can play with different factorings of responsibilities into capabilities and onto elements. And we can evaluate our postulated alternatives with respect to coverage or completeness, how they address system challenges, as well as balance, harmony, conceptual integrity and consistency, in addition to the achievement of desired system outcomes (capabilities and properties). All of which allows for a much more active, adaptive, creative exploration of the system concept and structural organization before the direction is channeled and cast in shaping, anchoring expectations and code.

We don't claim to be able to so well conceive of the system in its ultimate form that we go into system development with a perfect and complete design. We do claim that we can begin the evolutionary adaptation process with a more plastic and malleable medium than a growing mass of code. And then we can support code evolution with, again, a more malleable medium for exploring architectural-level refactoring and adaptations -- changing sketches and more formal models to explore the impact of ideas before we incur the cost of making the changes in the code base. All the while maintaining greater intellectual traction over a complex system, because we have cognitive aids to system understanding and work with system constructs in their own terms, rather than always and only in terms of code which contains details that obfuscate. 

How: Some Comments on Creating and Evolving the Conceptual Architecture

During early system conceptualization, we start to envision the form or shape of the system, its boundaries and interactions, its primary elements and their interactions. The sketchy shapes of these elements, expressed mainly in terms of their responsibilities, analogies, and drawing on patterns and experience, take a rough form, allowing us to investigate the system concept. Then, as the system concept (identity, value propositions and capabilities) takes shape, so too does the Conceptual Architecture.

In designing complex systems, separation of concerns is a key strategy for gaining intellectual traction and enabling ourselves to reason about important design decisions while considering emergent properties that are part and parcel of the decomposition-composition dual of systems design and development. The concerns focused on at this point are related to giving shape to the system paying attention to user-facing capabilities, production/operations-facing challenges and developer-facing properties of the system.

The architectural elements of software systems (that is, elements significant enough to the system to draw out and deal with in architectural design) are constructs of inventive human thought. We mold and shape them to accomplish a purpose, subject to constraints and forces. We leverage analogies to draw experience and knowledge from other domains, including mechanisms of nature (e.g., swarms), man-made (e.g., mechanical mechanisms like hubs), and social constructs (e.g., social and economic mechanisms like brokers).

The Conceptual Architecture Diagram is a lightweight and yet powerful tool for sketch-prototyping the system structure, and for rendering the structure of an existing system, to explore adaptations to it and as an aid to system understanding. Thus it allows us to explore the system from its earliest conception through evolution. As we experiment on paper, it is cheap to postulate various alternatives and explore their ramifications.

As system capabilities are explored, the associated responsibilities are allocated to "chunks" or components of the system, and these responsibilities are an important thinking tool for architectural designers who need to make software abstractions cognitively malleable and communicable. With something so simple as elements each with their list of responsibilities and relationships, we can experiment with different factorings and hence different abstractions or elements each with a different basis for cohesion of responsibilities, balancing forces, and how each resolves inherent tradeoffs. Further, it becomes possible to assess:

• clarity of responsibilities

• whether responsibilities are missing

• components for cohesion of responsibilities

• the system shape for balanced distribution of responsibilities

Architectural components may play different roles in the system, so their responsibilities are related within role-scopes. That is to say, for example, a component may play a minor role in system health monitoring and an active role with respect to some other system capability (e.g., providing shopping cart functionality).   

In highly dynamic systems, as software intensive systems are, structure and dynamics are designed together -- if we focus on structure we impact dynamics, and vice versa. When we focus on dynamics -- on behavior and "how it works" and properties that emerge from the structures as they perform their function and interact -- our design decisions have implications for, and need to be expressed in terms of, the structure for the structure houses, if you like, the implied responsibilities and interfaces. So we explore behavior, in our mind's eye and with responsibility lists, or on paper or the white board, using rough diagrams or informal sketches that just sketchily map interactions to accomplish some piece of behavior. During early conceptualization we're careful not to distract ourselves with model precision; we're trying to clear some of the fog of uncertainty. We're attending to our need to move fast and acknowledging that we need to "see" and think "out loud" "in pairs" or small groups, and to "get our hands dirty" with the cheapest possible design medium -- including sketch prototypes and code experiments. But importantly, using diagrams to create a shared thought-space to learn and explore just enough to discover and resolve the significant uncertainties and challenges of this moment. And using CRC-Rs and informal descriptions to capture the design thinking (not just the outcome in terms of structure, but the reasoning, including assumptions we're making), so it can be improved and shared. Then, as we progress with iteratively and incrementally building the system, our Conceptual Architecture diagrams and explanations mature, and continue to be important thinking tools as we evolve the design.

The conceptualization of a product isn't limited to the conceptualization of its use, but the product in use. We're designing what the product is capable of in conjunction with what the product is and how it is built. A car isn't a faster carriage, and although early cars had aspects of a carriage, the engine was quite different than a horse, and this translated to mechanisms to make the wheels move (from passive to active). In other words, when we conceptualize a new kind of product we iterate across designing capabilities and the mechanisms and structures that deliver those capabilities, and the product changes the ecosystem (a gasoline powered car raises the need for places to replenish fuel) so to make the product successful we also (impact the) redesign (of) key aspects of the ecosystem or larger system of systems into which our system fits, interacts and in important ways shapes and is shaped by.

“In most people’s vocabularies, design means veneer. It’s interior decorating. It’s the fabric of the curtains and the sofa. But to me, nothing could be further from the meaning of design. Design is the fundamental soul of a man-made creation that ends up expressing itself in successive outer layers of the product or service. ... The essence of the iMac is to be the finest possible consumer computer in which each element plays together" -- Steve Jobs

Conceptual Architecture grants us passage between the from-the-outside and the from-the-inside views of the system, allowing us to design interactively across the concept of the system and the internal concepts that allow that conception to take form. And it allows us to think about the elements and how they play together. More specifically, it is the conception and narration of the organization of the parts or elements of the system and its key, or architecturally significant, mechanisms -- as the system is conceived, developed and evolved. More pivotally, it is an agent of consilience, helping us to bridge various divides in system design.

But again I must emphasize, Conceptual Architecture is not a write-once read-only (I carbon stamp myself with the allusion, huh?) sort of thing. It is useful early, and throughout the evolution of the system, serving as an aid to thinking about and communicating system structure, evaluating alternatives, designing and redesigning as we create and evolve the system.

When: Early, and then often

The question of when has been implied in the earlier sections, but to make the point explicit: we work on drafts of conceptual architecture early. Very early! This is a cheap way to conduct thought experiments about the system and how it will work and be organized. We can think of conceptual architecture as sketch-prototyping the system, and create and evaluate different architectural alternatives. Hence, "early."

As the system matures, the conceptual architecture continues to play that role -- a vehicle for sketch prototyping conceived changes. But it also shifts to a useful vehicle for system understanding, which behooves us to keep the conceptual architecture view(s) up-to-date as we evolve the system. Hence, "then often."

For an existing system whose architecture documentation has become out-of-date, this is a crucial view of the overall system. Static code analyzers can help build up a view of components and dependencies. Even more powerful, is to bring the "system in the room," so to speak -- that is, bringing together developers (and for larger systems, tech leads) responsible for significant system elements -- to run through key use cases and "draw out" (literally and figuratively) the architecture diagram and accompanying CRCs. This is, in effect, playing the old CRC role playing game, but capturing existing responsibilities and staying at the architectural component (microservice, module, ...) level. Architecturally significant mechanisms can be sketched out in this context, or in the responsible subteams. But the value to getting the "tribal memory" that exists in pockets and teams into a shared view, is enormous.

Conclusion

In summary then, Conceptual Architecture supports the design and evolution of the system, allowing us to pay attention to system level concerns, creating enough order at the system level to provide cognitive traction, while allowing for organic emergence of a natural order to the system.

* Identifying responsibilities in simple lists like this enables early factoring and refactoring.

Acknowledgments

Thanks to Stuart Boardman and Charlie Alfred for comments. I have integrated some responses, with more work still to be done. Thanks also to Michael Hill for pointing out the need for the When section!