Accidental Complexity and Embracing YAGNI

As a software engineer, I am a self-described minimalist.  My goal is generally to seek the simplest possible technology and architectural solution with the least amount of moving parts to build, maintain, and extend.  Often times, this manifests as solutions that are “simple and stupid”.  I like to think of a well-architected piece of software like designing a satellite: a satellite with 10 moving parts will always be more ideal than one with 20 moving parts because it has 10 less points of failure.  Did we really need those 10 additional moving parts?

Scott Carey of InfoWorld recently published an essay that touched a nerve: Complexity is killing software developers

“Complexity kills,” Lotus Notes creator and Microsoft veteran Ray Ozzie famously wrote in a 2005 internal memo. “It sucks the life out of developers; it makes products difficult to plan, build, and test; it introduces security challenges; and it causes user and administrator frustration.”

If Ozzie thought things were complicated back then, you can’t help but wonder what he would make of the complexity software developers face in the cloud-native era.

Justin Etheredge, cofounder of the software agency Simple Thread, helpfully differentiates between essential and accidental complexity. He told InfoWorld, “Essential is the complexity in the business domain you are working in, the fact that enterprises are extremely complicated environments, so the problems they are trying to solve are inherently complex. The other area is accidental; this is the complexity that comes with our tooling and what we layer on top when solving a problem.”

The cloud-native era has ushered in the potential for more accidental complexity than ever before, setting a collision course between developers, who want to leverage the full toolkit available to them, and their bosses, who want them to focus on delivering value to customers.

Martin Fowler has previously written about the acronym of YAGNI or “You Aren’t Gonna Need It”, a term coined by Kent Beck, which I think aligns with this idea of “accidental complexity” we see nowadays.

The term can be applied at both a product level as well as a architectural level and Fowler makes a clear distinction:

Now we understand why yagni is important we can dig into a common confusion about yagni. Yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify. Yagni is only a viable strategy if the code is easy to change, so expending effort on refactoring isn’t a violation of yagni because refactoring makes the code more malleable. …[I]f you do have a malleable code base, then yagni reinforces that flexibility. Yagni has the curious property that it is both enabled by and enables evolutionary design.

I also argue that yagni only applies when you introduce extra complexity now that you won’t take advantage of until later. If you do something for a future need that doesn’t actually increase the complexity of the software, then there’s no reason to invoke yagni.

These days, when I see an architecture for an enterprise app with Kubernetes and Docker, the first thought that pops into my head is YAGNI.

I’m a big fan of Gregor Hohpe’s take on Serverless.  I tend to think that most teams would be more productive the further they move towards serverless solutions where the underlying runtime is managed by Amazon, Microsoft, or Google.

The reason is that business value is rarely manifest in the underlying infrastructure capabilities and in most cases, “you aren’t gonna need it” when it comes to the fine grained control over the runtime offered by operating your own layers of infrastructure these days (even if it is infrastructure as code).  I think that this is especially true in the enterprise space where applications are still, by and large, just fancy spreadsheets and/or file shares.  The further a team can move away from thinking about the underlying runtime, the more productive they will be in building and delivering actual business value, faster.

A second aspect of this is that each layer adds complexity;  it becomes another runtime, another piece that needs to be updated, understood, owned, transferred, deployed, and maintained.

In the real world, we’ve seen the same kind of shift all around us.  Restaurants have shed their delivery drivers and rely on Uber Eats and Doordash for “delivery as a service”.  Whereas one might have rented a car for business travel in the past, Uber and Lyft give us “transportation as a service”.  Each layer removed creates efficiencies, reduces the operating complexity for the consumer of that service, and adds convenience.

Complexity vs Innovation

In my experience, innovation and creation of value tends to exist in the space left over from having low complexity because a team of a given level of skill and experience will have more time and intellectual capacity to dedicate to creating high value, innovative solutions (A).  This is also why the start of a project always feels so fresh: there’s no debt and no complexity that you’re pushing against when you start from a clean slate.

When a team is well matched to the complexity of the technology and architecture, it leaves little space for innovation and creation of value since the effort of the team is dedicated to building and maintaining.  In such cases, it is necessary to have bigger teams to realize the innovation (B).

On the other hand, when the complexity is higher than the skill and experience of the team, this creates dreaded tech debt because nothing is ever done well, nothing is ever done cleanly, nothing is ever distilled down to the simplest manifestation, nothing is ever refactored or it’s refactored poorly or refactored partly (C).  Because the complexity is higher than the combined skill and experience of the team, it makes it harder to innovate because so much energy, intellectual capacity, and time is spent holding back the unnecessary complexity.

Complexity creates high cognitive load for even small changes.  It creates drag and slows teams down and only accumulates tech debt until the team, product, or company is no longer competitive because of the lack of innovation.  Complexity costs more because you need more skill and experience to be productive when the complexity is high.  Each additional layer of the stack that an engineer has to interact with is another layer of competency, cost, and complexity that is necessary to create value and innovate.

As Fowler states, “[YAGNI] does not apply to effort to make the software easier to modify“; the effort invested in making software more agile, more malleable, more flexible, and faster to adapt usually means cleverly designing away complexity and intentionally building solutions that require low cognitive load to build, deploy, operate, maintain, and extend.

That left over skill, experience, and capacity is the space where innovation and value manifest.