The Page That Fired Every Morning Was Correct

Every morning around 7:30, my phone lit up: Coquina down. I'd check. Coquina was fine. The memory service was up, serving reads, answering queries. And every morning the health-check worker had already fired off a page and triggered the self-heal runbook — restarting a service that didn't need restarting.

The worker wasn't lying. It was asking one question — did /api/health answer within five seconds — and the answer was honestly no. Under morning Ollama contention, the health endpoint blocked long enough to time out. The worker saw the timeout, mapped it to status="error", published health.degraded, and paged. Every step of that chain did exactly what it was told. The problem was that "didn't answer fast enough" and "is actually down" are not the same thing, and the worker had one bucket for both.

One endpoint is one question

A health endpoint that does readiness checks — Postgres connected, Chroma responding — takes real time to answer. That's the point of it. But when something else on the box is leaning on the GPU, "takes real time" becomes "takes more than five seconds," and a slow readiness check is indistinguishable from a dead service. The worker couldn't tell the difference because it only asked one question. If /api/health didn't answer, Coquina was down. That was the entire model.

Slow is not down, but it's not fine either

The fix was a second question. When /api/health times out, the worker now hits /api/live — a liveness probe, not a readiness check. Any HTTP response from that endpoint, even a 404, proves the daemon is serving. The worker classifies three outcomes instead of two: ok or degraded means the full health check answered and the status is whatever it reported; slow means the health check timed out but the liveness probe proved the service is alive; error means both failed.

degraded — a real Postgres or Chroma problem — pages immediately. error — truly unreachable — pages immediately, self-heal intact. slow gets a two-strike gate: the first one is suppressed, and if the next check is also slow, it escalates to degraded and pages. A service that's briefly contended gets a pass. A service that stays contended gets attention.

The gate had to be leak-free

The two-strike gate sounds simple until you consider what happens on strike one when something else is also broken. Say Coquina is slow and Ollama is genuinely down in the same health cycle. The worker should page for Ollama — that's real — but it should not include Coquina in that page, because Coquina's strike-one slow is suppressed. If the suppressed Coquina status leaks into the page or into the self-heal's check dictionary, the runbook sees a Coquina error that isn't there and restarts a service that's fine.

So a suppressed strike-one slow is rewritten to ok before it reaches anything downstream. The self-heal runbook never sees it. A co-occurring real failure pages on its own merits, and Coquina is not in it. The gate doesn't just suppress the page — it scrubs the evidence so nothing downstream can act on a status that was never confirmed.

The false page was the easy symptom

The tempting fix was a longer timeout. Give /api/health ten seconds instead of five, and the morning contention window probably passes. But a longer timeout is a bet that contention will always be shorter than the timeout, and that bet loses the first time Ollama loads a larger model or two workers hit the GPU at once. The timeout isn't the problem. The single-question model is the problem. A health check that can only say "answered" or "didn't answer" will always conflate slow with dead, no matter where you set the threshold.

Twelve tests for a bug that looked like one if statement

The fix is one file and one test file. The test file has twelve cases — not because the logic is complex, but because the interactions are. Strike one with no co-occurring failure. Strike one with a co-occurring Ollama failure. Immediate page on error. Immediate page on readiness degraded. Recovery resetting the strike counter. The liveness probe returning a 404 and still counting as alive. Each of those is a scenario where the old code did the wrong thing or where the new code could silently regress.

The unit that matters most is the co-occurrence test: Coquina slow, Ollama down, and the page names Ollama but not Coquina. That one test encodes the entire reason the gate exists. Without it, someone — me, a future refactor, a model writing code at 3 AM — could simplify the gate back to a naive suppression that leaks, and the test suite would stay green.

A system that monitors itself has to be at least as careful about what it says as about what it sees. The health check was never missing a failure — it was reporting one that wasn't there, every morning, with complete confidence. The fix wasn't better detection. It was a second question and the discipline to stay quiet until both answers agreed.