Half My Proof Wasn't Runnable

Docs rot. That's not a new observation, and the usual response — "keep them updated!" — has never worked for anyone, including me. My own repo is the evidence: the root ARCHITECTURE.md is an unfilled template nobody ever completed, docs/GIT_WORKFLOW.md describes a workflow I don't use, and my CLAUDE.md carries at least six facts I know are stale and haven't fixed.

So when I built a skill library — 14 skills for homelab, 14 more for Coquina, on top of the global set — I knew exactly what I was building. A rot machine with a fresh coat of paint. Worse than the docs I already had, actually, because every file would open with a line saying "Verified against repo state 2026-07-06." It wouldn't just rot. It would rot credibly.

The idea I had against that was simple enough to state in a table. Every skill ends with a Provenance section, and every volatile fact in it gets a row:

Fact	Re-verify	Drift looks like

A claim, a one-line read-only command that proves the claim, and a description of what it looks like when the claim goes bad. 181 rows at build time. Docs that carry their own proof.

Then I wrote the worker that actually runs them, and it told me something I didn't want to hear.

Ninety-six

The freshness worker parses the Provenance tables, executes each re-verify command under a hardened read-only allowlist, compares the result, and files drift as work. Before running it for real, I checked how many of my existing rows it could even execute.

96 of 183 were unrunnable.

Not wrong. Not stale. Unrunnable — they could not be executed at all. Commands with cd chains the sandbox rejects. A python invocation that a substring filter can't prove is read-only. Rows whose "command" was really a sentence describing what you might type. One that referenced an environment variable — GRAIN_REDIS_PASSWORD — that doesn't exist anywhere in my system, so it would have silently resolved to an empty password and connected to nothing, and reported that as fine.

More than half of my proof was decoration. And I want to be clear that this wasn't sloppiness at the margins — those rows were written deliberately, by me and by a model working carefully, in the specific act of being rigorous. Writing a re-verify command felt like proving something. It produced the exact sensation of having done the work. The rows looked like proof, they were shaped like proof, they sat in a column headed "Re-verify," and 52% of them would not run.

That's the finding I keep coming back to. The feeling of rigor and the fact of rigor are different things, and only one of them survives contact with an executor. I could not tell them apart by reading. Nobody can. The only thing that separated the real rows from the theatrical ones was a machine that tried to run all 183 and didn't care how sincere any of them were.

Fixing them was a docs-only PR that touched no code: fold the cd chains into paths, convert the python and sed counts into grep -cE '…' (expect N), and — this is the part I'm proudest of — adjust the wording of the Fact and Drift columns "to honestly describe what the proxy proves." Because a grep count is not the same claim as the original sentence. If the command can only prove something narrower, the fact has to shrink to match it.

Result: 2 of 184 unrunnable. The two survivors are memory queries no shell command can express, and they're marked manual on purpose rather than faked into something executable.

The worker files tickets, not warnings

The loop only counts if drift becomes work. So the freshness worker doesn't log, and — deliberately — doesn't post to Slack. I learned that one the hard way: an earlier incident taught me that adding one more thing to an alert channel is how you train yourself to ignore the channel.

Instead it files an Asana task, one per drifted skill, deduplicated against what's already open: [Skill-Drift] <skill>: N facts stale. It runs Sunday at 07:30 — chosen to sit outside the 5 AM chain window, after the 06:45 watchdog, clear of the ~3 AM Sunday router reset — with max_retries: 0, because re-running a scan that failed changes nothing and I'd rather it fail loudly once.

I opened my board tonight and there were four of them waiting: homelab-tdd, homelab-parallel-dispatch, homelab-failure-archaeology, homelab-change-control. Six stale facts between them.

Not one is a real drift. Every single finding reads unrunnable. Three are query_memory calls — the two documented rows no shell command can express, plus one more — and three are rows where the re-verify cell is simply empty, a fact with no proof attached at all. So the verifier is filing tickets about the verifier, which is a small joke, and also the honest state of things: two of those skills were written after the normalization pass and never got held to it, so they arrived carrying exactly the decorative rows I'd just spent a PR removing everywhere else.

That's what a maintenance loop actually looks like from the inside. Not a system that catches drift — I haven't had a real drift yet — but a system that keeps noticing the places where I've stopped being able to prove things, and refuses to let me forget them by putting them on a board instead of in a log.

The allowlist that runs those commands is deliberately paranoid, and each rule is a scar. No python3 -c, because a substring filter cannot make an arbitrary program read-only. The $ character rejected outright, because ${HOME} is a read path into a parent environment that might hold secrets. curl restricted to exact localhost, with --resolve and -x rejected so nothing can be redirected outward. Every command runs under a scrubbed environment — PATH and HOME only, never my Asana token — and output is secret-redacted before it's stored anywhere.

The refusal that made me trust it

The worker carries two separate ways of declining to say "fine," and I want to be exact about which one has actually done any work, because I nearly wasn't.

The first is a fraction gate: if more than half a scan's rows come back unverified — unrunnable, timed out, or still ambiguous — the run is marked with concerns instead of completed. The comment explains the reasoning: reporting success on an unverifiable scan "repeats the timeout-as-success anti-pattern," a reference to an earlier incident where a tool silently no-op'd and my chain scored it as a pass.

That gate has never fired. It's insurance.

The one that fired is smaller and less clever. On July 12th, the local model doing ambiguity triage died mid-scan, and the exception handler turned the failure into a concern rather than swallowing it:

done_with_concerns — triage failed (qwen3:8b): triage LLM call failed:
                     Ollama call failed — 47 rows remain ambiguous

One done_with_concerns against three completed in the log. And the scan was, by any reasonable summary, going fine — 196 facts, 143 fresh, zero drifted, 47 ambiguous. Fifty-three couldn't be verified out of 196, which is 27%, nowhere near the 50% threshold. The fraction gate wouldn't have caught this. Every headline number was green.

It reported concerns anyway, because a component it depended on had died, and it wasn't going to hand me a summary that quietly rested on 47 questions it never got to ask.

That's the distinction I keep having to relearn. The impressive-sounding safeguard — the threshold, the ratio, the gate with a constant at the top of the file — is the one that's never mattered. What mattered was an exception handler that appended a sentence instead of passing.

Put that next to the other half of my infrastructure. My figma-live MCP server has been reporting healthy since June. The health check behind that word runs checkCommand('npx'). It confirms that npx is on my PATH. It has never once contacted the websocket the server actually depends on, never validated the package, never proved a single thing about whether the integration works — and it says healthy, in green, every time I look.

Two subsystems. One says done_with_concerns when it couldn't check 47 rows. The other says healthy because a binary exists. Only one of them has ever told me something I didn't already want to hear.

The library that gets audited for lying

The build itself was a fan-out — 11 discovery readers burning 1.59M tokens across 448 tool calls, 14 authors writing in parallel, a three-lens review, a fixer. Out came 5,778 lines, 621 canonical facts, 62 explicitly labeled uncertainties. The review caught 31 findings and the fixer applied 16 of 16 blocking and important ones.

The catches I liked best went in both directions. One author was handed a discovery note claiming "worktrees have no hooks," went and checked, and refuted it live — linked worktrees inherit the main clone's hooks; only fresh clones are bare. But another author got skeptical about a tally from an old damage report and declared it unverifiable, and the reviewer found the number sitting verbatim in the source file and put it back. Skepticism caught a false claim. Correction caught a false skepticism. You need both legs, and the second one is the one people forget.

The number I keep returning to, though, is 96 of 183. Not because it's embarrassing — it is — but because of what it implies about everything I haven't pointed an executor at. That library is the only thing in my house that gets mechanically audited for lying. It got audited within 24 hours of being built, and the audit found it lying, at scale, in its most rigorous-looking column.

Everything else I own is still on the honor system. I have no reason to believe it's doing better. I just haven't built the thing that would tell me.