HOME / BLOG / Every Step Was Correct and …
MARKDOWN

> Every Step Was Correct and the System Still Broke

|
~5min READ
debugging systems observability composition measurement postmortem
// A year of bugs on my own site and services, sorted by how they were found. The expensive ones share a shape: no individual decision was wrong, nothing logged an error, and per-step review could not have caught any of them.

Most debugging advice assumes a wrong step exists and the job is finding it. The bugs that cost me the most this year did not have one.

Each of these was live for weeks or months. Each would have passed a review of every file involved, because every file was right. What was wrong was the space between them.

The shape

Here is the clearest instance. A rate limiter for LLM traffic has to hold budget for output it cannot measure yet, so it reserves an estimate and reconciles later:

reserve   hold budget for a request whose output length is unknown
commit    reconcile against what generation actually consumed

Two phases, both obviously correct. Now a client crashes between them.

Deducting the reservation was correct. Not refunding — no commit arrived — was also correct. So the hold stays. Forever. The tenant's effective quota decays monotonically, one crash at a time, and converges on zero.

At which point they can submit nothing, and there is no error to look up. Not a warning, not a failed assertion. Every decision in the chain was the right decision.

The fix is a third phase, expire, and the reason it is not optional is that a lease is the only thing that distinguishes "crashed" from "still running". Without it those two states are byte-identical on the ledger.

Lesson: when a system degrades monotonically while every step audits clean, look for an allocation with no symmetric reclaim path, where the counterparty is allowed to vanish.

Same shape, five more times

A variable read under one name and exported under another. settings.py read DJANGO_DEBUG. The deployment exported DEBUG. Both names are reasonable; neither file is wrong; no tool compares them. The app ran with DEBUG=True in production for months. Because DEBUG was then true at import, the SECRET_KEY guard did not fire either, so it generated a random key per process — which does not raise, it just logs people out at random.

A health check that had never passed. The container was created with a hand-written docker run, which mangled the quotes in the probe into a literal \x27. Guaranteed SyntaxError, every 30 seconds, for the life of the container — tens of thousands of failures. Docker dutifully recorded them. Nobody reads FailingStreak on a container that is obviously up.

Headers that vanish by inheritance. In nginx, a location block that declares any add_header silently discards all inherited ones from the server block. Five locations each added a sensible Cache-Control, and in doing so dropped nosniff from every user upload. Each block is correct read alone. The composition is the bug, and it is invisible unless you request each path and look at what actually came back.

Markup written against a stylesheet that was never loaded. Every template here is authored in Tailwind utility classes. Tailwind is not loaded; what exists is a hand-written partial subset. An audit found 190 class names used and defined nowhere — text-xs in 83 places, mt-auto, ml-auto, blur-[100px]. Nothing errors. Card subtitles just quietly render at body size, footers quietly do not push down. The page looks slightly wrong everywhere, which is exactly the hardest thing to get a bug report about.

18 MB downloaded per page view, for nothing. base.html preloaded an unsubsetted CJK font. styles.css names that family first in three stacks. The @font-face that would load it lives in a stylesheet linked from nowhere. Three files, each defensible; the font could never apply, and every visitor paid for it anyway.

Why per-step review cannot find these

A review asks "is this correct?" of one thing at a time. All of these pass that question. They fail a different one: "when these are composed, what actually comes out?"

Which suggests the only check that reliably works — stop reading inputs, read outputs.

  • Don't read the template; grep the rendered HTML. I have a note on this site about multi-line {# #} being rendered to visitors rather than treated as a comment. While writing this round of changes I did it twice more, once in base.html, where it appeared on every page. The sweep caught both. Reading my own diff did not.
  • Don't read the nginx config; request each path and dump the headers.
  • Don't trust publish: true; run the loader's dry-run and read the create: list. Eight notes here had been flagged for publication and never deployed. The export is opt-in, the deploy is a separate manual step, and nothing connects them.
  • Don't quote a benchmark's peak; report where the spread closes. The gateway above serves 2,999 of 3,000 offered requests, and the capacity I report is 1,000 — above ~2,000 the p99 moves more between two runs of the same binary than it does across the entire range below. That number stopped describing the service and started describing a busy host.

The uncomfortable part

I found most of these while looking for something else. The font was found while fixing colour contrast. The health check was found while making the container run as non-root. The 190 missing CSS classes were found because a heading and its subtitle looked 2px apart.

I do not have a method that finds this class on purpose. What I have is a habit: after any change, check the artifact rather than the source — the rendered page, the response headers, the dry-run output, the spread column. Every one of these was visible there, immediately, and in no other place.

Related notes: a hold without a lease decays to zero · grep the rendered output · tools that silently do less · report capacity where the spread closes

BACK_TO_BLOG()