Software That Refuses to Lie

One of my monitoring dashboards once showed "all clear" while roughly 4,278 tasks sat in five queues it wasn't even watching.

The bug was quiet. That's what made it dangerous. The queue-depth code hardcoded a handful of queues — somewhere between two and six of about forty — instead of enumerating them all. Everything it checked was empty, so it drew green. The fire was entirely in the part it didn't check.

I've written here about software that watches itself — the kind that catches its own drift and heals it. This is the other half, and the harder one: what a system does about the things it can't watch. The dashboard above couldn't see five queues, and reported green anyway.

A dashboard that's wrong loudly is survivable. One that's wrong quietly is not.

Think about who a dashboard is actually for. Its entire purpose is to raise an alarm I'd otherwise miss. That's the contract.

A dashboard that goes blank, or throws an error, breaks that contract in a way you can see. You look closer. You go find out what's wrong. The failure advertises itself.

A dashboard that stays green while the system burns breaks the same contract silently — and worse, it suppresses the very alarm it exists to raise. It doesn't just fail to help. It talks you out of looking.

The fix wasn't a nicer chart. It was to stop trusting a curated list and enumerate every queue in the namespace — scan the whole space, count all of it. If a queue exists, it gets counted.

I could have just added the five missing queues to the hardcoded list. That's the ten-minute fix, and it would have been the wrong one. It patches this instance and leaves the disease: the next queue anyone adds is invisible again, and the dashboard lies again, and nobody knows until another few thousand tasks pile up. Enumerate-everything is more code, and it's the only version that stays honest as the system grows. That tradeoff — cheap-and-brittle versus more-work-and-durable — is the whole job.

The second time, it was a ghost.

Same disease, different system. A nightly chain wrote its live run-state with a short expiry and deleted the record on completion. So the 5 AM run had, by the time I looked at 7, simply vanished. It finished successfully, then erased itself.

Any "here's what happened since you last looked" view now had a problem: the thing it needed to report on was gone. And a view that has to report on something it can't read has exactly two options. It can say "I don't know." Or it can guess.

Guessing, in instrumentation, is just fabrication with better manners.

The fix was boring and correct: write a durable run record that never expires, so a run that finished at 5 is still fully readable at 7, or at noon, or tomorrow. Make the truth outlive the moment it happened.

If I can't prove it, I don't paint it green.

That's the load-bearing rule now, across the whole dashboard. Call it the honesty gate. Anything the backend cannot actually read renders as "unknown," or "no runs" — never as a confident all-green stitched together from data that isn't there.

This is a design decision before it's an engineering one, and it cuts against the instinct of every dashboard I've ever built. Green feels finished. Green feels like good news. An empty state that says "unknown" looks, at a glance, like something's broken.

But "unknown" is the honest signal, and honest is the entire point. Rendering "unknown" is a feature — the interface telling you the truth about its own blind spots instead of papering over them with a color that means relax.

Green is a claim. A claim you can't back is a lie you'll act on.

That's the part worth sitting with. "Everything's fine" is not a neutral default. It's an assertion. And an assertion you can't support with evidence isn't a harmless placeholder — it's a lie, and the specific danger of this one is that you'll believe it and stop looking at the thing that's actually broken.

I applied the same rule somewhere it looks unrelated: a weekly worker that re-checks whether my documentation is still runnable. If more than half of a document's checks come back unverified — or if zero come back confirmed while checks clearly exist — it doesn't report success. It reports "done with concerns." It refuses to claim a green it can't stand behind.

It's the same reflex as reviewing code by handing it to a second model whose only job is to break it, and the same reflex as an audit where no finding counts until it survives a skeptic: trust isn't a system that always says things are fine, it's a system that tells you when it doesn't know.

The most trustworthy dashboard I own is the one most willing to admit it can't see.