Perspective

Most teams are troubleshooting yesterday’s environment

Cloud environments change faster than anyone’s understanding of them. The result is teams making security and architecture decisions against a picture of a system that no longer exists, and calling the documentation that would have prevented it bureaucracy.

·

9 min

var(--variable-rpcX8dOAk)

Key findings

In environments where infrastructure changes land within days, the interval between understanding a system and that system changing is shorter than the interval between decisions made about it.

The hardest problems in cloud engineering are not technical. Almost every technical problem has a known path forward. Knowing who owns what, and why something was built the way it was, does not.

An environment that works only because a few people remember how it fits together is not a mature environment. It is a functioning one with a single point of failure that appears on no diagram.

Legibility precedes change, and change precedes modernization. That order does not reverse, and attempts to reverse it produce either an outage or a second system running alongside the first.

In a cloud environment, infrastructure changes faster than anyone’s understanding of it.

Someone adds a VM. Someone else adds a networking rule, or an identity, or an integration, or a security control. Individually each change is small, and each one is justified. Collectively, within weeks, the environment has moved and the architecture diagram has not. The inventory is stale. The assessment that informed the last decision describes a system that no longer exists.

Then a real question arrives. A security decision, a compliance obligation, an architecture change. And it gets answered against the old picture.

Not deliberately. Nobody decides to use outdated information. It is simply the information that is available, and re-establishing the current state takes time the schedule does not have.

The result is that people end up troubleshooting yesterday’s environment instead of today’s.

Why it keeps happening

The obvious explanation is carelessness, and it is usually wrong. The teams where this shows up most clearly are not careless. They are fast.

Speed is the cause. In environments where significant infrastructure changes can land within days, the interval between “we understood this system” and “this system has changed” is shorter than the interval between decisions made about it. Re-establishing the current state before each one feels like friction, because in the moment that is exactly what it is. A day spent producing something that looks like documentation rather than progress.

So it gets skipped. And the cost arrives later, somewhere else, attached to a different problem.

That displacement is what makes the pattern hard to learn from. An incident is rarely traced back to an assessment that was six weeks out of date when someone relied on it. It gets attributed to the change that triggered it, a fix is applied, and the underlying condition survives untouched.

What the drift actually costs

Three things, in rough order of how often we see them.

Decisions made against a system that no longer matches. A control is designed for a topology that has since changed. An exception is granted based on an assumption that is no longer true. A risk assessment describes a boundary that has moved.

Changes whose blast radius nobody can establish. When the dependency map is informal, every change becomes a judgment call made under uncertainty, and the rational response to uncertainty is to avoid changing anything. That is how environments freeze.

Work repeated because the prior work is not discoverable. Someone solves a problem that was solved eighteen months ago by a person who has left, because there is no record that it was solved or how.

None of these present as documentation failures. They present as engineering problems, security findings and schedule overruns.

The position

Before a major security, compliance or architecture decision, re-establish the current state. Not the state as documented. The state as it is.

That sounds simple and it is not, because it means accepting that knowing what you have is continuous work rather than occasional work, and that it belongs inside the engineering rather than in a report produced at the end of it.

The harder problem is not technical

The hardest part of cloud engineering is not the technology.

Most cloud problems are solvable technically. Azure, AWS, Terraform, Kubernetes, identity platforms, networking. There is almost always a technical path forward, and a competent engineer will find it. That part of the work is genuinely well understood, and it is the part the industry spends most of its attention on.

The harder problem is building an environment where people know who owns what, where changes are visible, where decisions are recorded, and where an engineer can understand why something was implemented six months after the person who built it has gone.

This is why we put weight on things some engineers regard as administrative. Tickets. Architecture documentation. Change control. Inventories. Runbooks. Handover documentation. Clear operational ownership.

We do not treat those as bureaucracy. We treat them as part of the engineering.

A cloud environment that works only because three engineers remember how everything fits together is not a mature environment. It is a functioning one with a single point of failure that does not appear on any diagram, does not show up in any risk register, and will not be discovered until one of those three people is unavailable.

The objection, and what we think it gets wrong

The strongest argument against this position is that documentation is always out of date, so maintaining it is a tax paid for something that does not deliver.

That objection is half right, and the half it gets right is important. Documentation maintained as a separate artefact, updated on a schedule by someone other than the person making changes, will drift. It always does. Producing it and then defending it as current is worse than not having it, because people rely on it.

What we think the objection gets wrong is the conclusion. The answer to drift is not abandonment. It is making the record a by-product of the change rather than a description of it.

Infrastructure as code is the clearest version of this. The definition of the environment and the environment itself are the same artefact, so they cannot disagree. A change that is not in the code does not exist, and a change in the code is self-documenting by construction.

The same principle applies further up. A ticket that is required to make a change is current by definition. An inventory generated from the platform’s own API is current by definition. A runbook tested during handover is current at least once.

The target is not perfect documentation. It is reducing the number of things that are true about an environment and recorded nowhere.

Things engineers call administrative

Treated as overhead

Actually load-bearing

Tickets slow the work down

Tickets slow the work down

Tickets are the record of why a change was made, for whoever inherits it

The diagram is out of date, but we all know the system

The diagram is out of date, but we all know the system

The system is known by three people, and that is a dependency rather than a strength

We will document it once it is stable

We will document it once it is stable

It is never stable, and the person who could document it has moved on

Inventory is a compliance exercise

Inventory is a compliance exercise

Inventory is what tells you which systems a change will touch

Change control is bureaucracy

Change control is bureaucracy

Change control is how a change stays traceable to a reason

Runbooks are an operations concern

Runbooks are an operations concern

A runbook is the difference between a handover and an abandonment

Handover happens at the end

Handover happens at the end

Handover is a property of how the work was done, not a phase after it

The sentence that tells you

There is a sentence that changes how we approach an engagement the moment we hear it.

We don’t really know why it was built this way, but we can’t change it.

That one line usually carries several things at once. There are probably undocumented dependencies. There is probably no reliable inventory. Knowledge likely sits with one person, or with a contractor who left. And the team is often not resistant to change so much as afraid of it, because nobody can establish the blast radius of touching anything.

The instinct at that point is to start fixing. That is the wrong move, and it is how a modernization project becomes an incident.

Reconstruct before you change

What we do instead is reconstruct. Six questions, in roughly this order:

What exists. An actual inventory, generated from the platform rather than from memory. Every subscription, every resource group, every identity, every integration.

What depends on what. Not the intended architecture. The observed one. What talks to what, on what ports, with what credentials, and what breaks if it stops.

Who owns it. For each component, a named person or team accountable for it. Where that cannot be answered, the answer is itself a finding.

What business process it supports. A system nobody can connect to a business outcome is either dead weight or something whose importance will only become visible during an outage.

What security or compliance requirements apply. Which controls touch it, which boundary it sits inside, what evidence its existence generates.

What happens if it fails. Not theoretically. Specifically, which processes stop and who notices first.

None of this is glamorous and none of it ships a feature. But it converts an environment from something a team is afraid of into something they can reason about.

The order does not reverse

Once an environment is understandable, it becomes changeable. Once it is changeable, it can actually be modernized.

That sequence does not run backwards. Every attempt we have seen to modernize a system nobody currently understands produced one of two outcomes. Either an outage, because a dependency nobody knew about was severed. Or a second system running alongside the first, because the old one could not be decommissioned with confidence, which is the same problem with more surface area and two sets of controls to evidence.

The uncomfortable implication is that the reconstruction work has to be funded, and it is the least attractive line item in any proposal. It produces no new capability. It makes nothing faster. What it produces is the ability to make the next decision on accurate information, which is impossible to price and obvious in hindsight.

We think it is the part of the work most worth paying for, and we say so to clients before they ask.

What this is

This is a position rather than a finding. It comes from working in environments where change outruns documentation, and it is offered as an argument to disagree with rather than as something measured.

We publish research with the method stated and the figures traceable to a named source. This is not that. It is what we think, and why.

Tell us what you are deciding.