Software Architects: Do We Need 'em?

Architectural design is system design. System design is contextual design — it is inherently about boundaries (what’s in, what’s out, what spans, what moves between), and about tradeoffs. It reshapes what is outside, just as it shapes what is inside.

Everyone who impacts the architecture (in effect makes architecture decisions — knowingly or not) is in deed an architect of that system. Questions such as this:

"Can modern agile software development support an individual Architect making important decisions on behalf of teams?" -- @obie

remind us that we need to reclarify the role of architects when complex systems are being design-evolved in lean and agile contexts. That is, in contexts where we recognize — and value — the role everyone plays in creating and evolving systems and their architecture. In what follows, we explore the need for a role that focuses on system design, requisite (system) cohesion and system integrity.

Agile Means Learning on Multiple Fronts

Agile in various of its flavors, feeding into frequent, if not continuous deployments, can raise the profile/visibility of the team and sense of urgency — we have, for example, "standups," "backlogs" and "release trains." These make progress, and hold-ups, visible; good, we can coordinate, but visibility can be pressure. We weight early and then frequent delivery to achieve shorter time to value, faster feedback loops. Good stuff.

Still, there is a "but" to get past: it takes strong discipline to keep a handle on code quality; to keep the code from falling into (massing, miring) technical debt. Not because we aren't good at our craft, but because we are learning what our system needs to be and become, and we are learning, upon reflection, if we pause to reflect, how to do things better, now that we know more what the system is becoming and how better to enable that (technically). And what strains the system, threatening its integrity (structure of the code and qualities like adaptability and all that is bundled therein, as well as qualities of the software in operation and use, including resilience, behavior under load and other issues of scaling, security and more).

"[Becoming] involves the transformation of potentialities into actualities" — David Ray Griffin 

Learning Means Changing, But Changing Takes Time

Having strong and organizationally-valued technical leadership throughout the project becomes more, not less, important in Agile (at least, beyond some point of organizational and technical complexity). The (structural and mechanism) design has to be continually advanced -- challenges surfaced, alternative approaches identified and evaluated, approaches tried out. This takes attention, so time. Time away from "moving forward." Ostensibly.

And Not Just Time. Experience and Ingenuity.

Software development is by nature development -- it moves the system into new territory. We have to create novel solutions to novel (permutations of) problems -- bringing experience, creativity, research and trying things out, much to bear on finding an approach that will address the challenges that arise. This ability to design, to solve hard problems, make tradeoffs, is a key attribute of the architect. Sure, others on the team have it too, but there's valuable experience that comes of seeing many things done, adding to the capability set to draw on, but also adding to a sense of what is risky, or hard. This helps in evaluating capabilities bundled into technologies we're evaluating for our stack, as well as in strategically sensing and evaluating where to build a capability to differentiate. There is a dimension of ingenuity and technical wisdom, experience and ability to see around corners, as it were, to identify the gotchas before they getcha. But there is also a dimension of organizational wisdom and effectiveness.

A Locus of (Over-arching) Responsibility for the System, as a System

Again, the team is participating in the design, impacting the architecture, making architecture decisions — but if architecture is everyone’s responsibility, then architecture is no-ONE’s responsibility. Someone (with accountability — and by that mostly I mean they hold themselves to account, managing their attention and responsibility) needs to be attending to the system as a system — one that exists in the context of other systems, with all of their respective forces and demands.

There's paying attention to the structural integrity of the code when that competes with the urge to deliver value and all we've been able to accomplish in terms of the responsiveness of continuous integration and deployment. We don't intend to let the code devolve as we respond to unfolding demands, but we have to intend not to, and that takes time — possibly time away from the next increment of user-perceived value.

There's watching for architecturally impactful, structurally and strategically significant decisions, and making sure they get the attention, reflection, expertise they require. Even when the team periodically takes stock, and spends time reflecting learning back into the design/code, the architect’s role is to facilitate, to nurture, the design integrity of the system as a system — as something coherent. Where coherence is about not just fit and function, but system properties. Non-trivial, mutually interacting properties that entail tradeoffs. Moreover, this coherence must be achieved across (microservice, or whatever, focused) teams. Across a lot of code. And then more code!

What It Takes

This takes technical know-how and know-when, and know-who to work with, to bring needed expertise to bear. And it takes technical wisdom, perspicacity, insight into not just consequences of design choices, but watching for design challenges ahead; for impacts across the system invisible to those whose perspective is shaped by working on a locally scoped piece of code (a feature, a service or microservice, anything bounded). Not just "laying down track in front of the train" but getting out ahead enough to scout out the lay of the land and know where we need to put in a bridge (or architectural mechanism) in readiness. (Which, by the way, is why we need to do some upfront architecting — just enough, which is a judgment call, drawing on experience and a combination of design sense and sensibility, timing and risk awareness.)

Leading as Noticing What's Not Being Done that Needs To Be, and Enabling

Can the team, in a self-organizing way, ensure that architecturally significant decisions get the attention they need? Anything that needs to be done, that is happening organically, or because someone else is leading (perhaps with mentoring), is wonderful, and to be encouraged and facilitated. 

The likely thing, though, is that dominant concerns of the day (this feature or microservice, now!) shape perception, which will mean playing catch-up on architectural concerns -- they'll be noticed when they bite hard. And architectural concerns are exactly those you want to anticipate, to the extent possible, given technical and organizational wisdom, experience, context awareness, perspective and imagination. (That is, they are, by nature, cross-cutting, have deep impact and high cost of change, are make-or-break structurally and/or strategically.)

Organizational Effectiveness , Not Just Technical Wisdom

Further, we shouldn't underplay the importance of technical and organizational gravitas, credibility, ability (and willingness, even desire) to lead, to create concert and coherence, pulling towards system integrity, not pulling apart in turf protecting battles. (For example: Want to simplify? Whose pet idea is going to be culled or redesigned? How will that be managed? Different — competing — ideas about how to accomplish something? Who makes the call? How? Stuck, because uncertainty and ambiguity in the fog of confounding change? Who's going to create "ground under the feet" by deciding, despite the risk of being "wrong"?

Hard things are hard. Harder when brain cells and emotions get tangled up in them. People are people, people! Leadership is about doing the hard things we aren't doing, that wouldn't be done anyway, as a matter of course because it takes gathering organizational will, determination, concert of minds and hands, to do. It's bigger than a person, or just a couple or three. It competes for attention with other things that are more "obvious" or near term or de facto part of the organizational habit and routine. And it's saying "no," sometimes — even when it's hard.

"I wasn't the one pushing things in the wrong direction, but I should have been the one to stop it." —  Chad Fowler  (around minute 23)

Technical Effectiveness , Not Just Organizational Wisdom

A team, and even more, team-of-teams, benefits much from a technical leader who deeply understands the technical ramifications and is advocating for the team and helping to shape the context (get resources, protect from political pressure, etc., but also do architectural investigations and design ahead of it becoming a trainwreck for the project, etc.) so the team can do great work, create a system with integrity -- structural and design integrity, but in the ethical sense too. Overly rigid dominance hierarchies that weren't a good fit for agility have left antibodies, but if we overcorrect in the direction of not valuing wisdom and foresight, experience, focus, context awareness, system-level problem finding-solving, ..., and leadership, other problems mount... But leadership in organizations is fractal, and fluidly, dynamically forming and reforming. Further, the architect (by title or simply by responsibility) leads where that is needed — including leading by the example of following well or stepping out of the way.

What we are paying attention to, shapes what we perceive and pay attention to.