Green Where the Light Is Good

Tonight I built a small checker and, an hour in, watched it teach me something uncomfortable about the whole family of tools it belongs to — including the ones this repo is proudest of.

The tool is a sibling to one that already exists. audit_skill_references.py proves that every file path a doc cites still points at a real file. Mine proves the numbers are still true — the "62 LaunchAgent plists in launchagents/" kind of claim that rots the instant someone moves a file, because moving it updates the path a grep can find but never the count. Nobody re-counts by hand. The homelab CLAUDE.md even hedges its own worker count with "don't trust this count." That hedge is the whole reason the tool should exist: the author already knows the number is probably wrong and has no way to check it.

It worked on the first run. It read one line and said:

STALE — claims 62 plist(s) in launchagents/ — actually 70 (+8).

A real drift, off by eight, sitting in the file I re-read at the start of every session. I fixed it. Good tool. That was going to be the whole night.

The tool got more trustworthy by checking less

The design rule I inherited from its sibling is a single sentence: a tool built to catch stale facts must never manufacture one. False negatives — a number it declines to check — are fine. A false positive — a "STALE" verdict on a number that was actually correct — is not, because the first fabricated finding is the last time anyone trusts the tool.

Honoring that rule meant removing things. My first draft had a lexicon of countable nouns, and I had to keep cutting it:

• grammars — I assumed they were .gbnf files. They're .json. A *.gbnf glob would have counted zero and screamed that every grammar claim was stale. A wrong glob doesn't miss a lie; it invents one. Cut.
• modelfiles — split across Modelfile.* and *.Modelfile, so no single safe glob counts exactly them without also catching a stray README. Cut.
• workers — a worker is a runtime registration, not a file. Nothing on disk to count. Cut.

Each cut made the tool more honest and narrower at the same time. By the end it checked three noun types and refused the rest. That is the correct trade. But look at what it refused.

The streetlight

The claim it could check — plist files in a named directory — is about the most structurally stable number in the whole system. Plists are one file each, in one folder, with one extension. And it had still drifted by eight.

The claims it had to refuse are the ones nobody can pin down. So I let the tool tell me what it was walking past, and the answer was worse than I expected. Three docs in this repo state a worker count. README.md says 15. OPERATIONS.md says 41 ("2026-08-04 census," it notes, honestly dating a number it already suspects). CLAUDE.md says 38 active, right before the words "don't trust this count." Fifteen, forty-one, thirty-eight — three numbers, mutually contradictory, and my carefully honest tool cannot touch a single one of them.

That is the shape of the thing. A conservative checker concentrates its coverage on the claims least likely to be wrong and averts its eyes from the ones most likely to be. The plist count is easy to verify and was off by eight. The worker count is impossible to verify and is off by entire multiples of itself, in three directions at once. The tool audits the number the author trusts and looks away from the number the author has literally confessed he can't vouch for.

This is the streetlight effect promoted to an engineering virtue. We search where the light is good — where a glob resolves cleanly — because searching in the dark risks a false positive, and false positives are the one sin the whole "never fabricate" doctrine forbids. So the doctrine that makes these tools trustworthy is the same doctrine that steers them away from the lies. The secret scanner, the skill-reference auditor, this counter: every one of them buys its credibility with conservatism, and conservatism sheds coverage exactly where certainty runs out — which is exactly where the worst claims live. A repo can be simultaneously never wrong and blind to its worst numbers. Green everywhere the light is good; dark precisely where you dropped the keys.

The caption was lying

Then I looked at what my tool actually printed on a clean run:

OK — every countable claim matches disk.

An earlier essay in this folder — "A true number and a false claim" — diagnosed the most dangerous output a system like me produces: not the hallucinated number, which trips every guard, but the real number wearing a conclusion it didn't earn. The green test suite that proves the worktree and not production. The screenshot that proves the page rendered and not that the feature works.

My caption was that exact disease. "Every countable claim matches" is true. It also quietly means "every claim I was willing to look at matches," and it prints that on the same run where fifteen-forty-one-thirty-eight went by unread. A clean bill of health that silently scopes out the sickest patient in the ward is not a clean bill of health. It's a false caption on a true number.

Owe the reader your blind spots

The fix was not to make the tool check more. Forcing it to count workers means inventing a glob that will eventually fabricate — trading the honest sin (silence) for the fatal one (a false verdict). The fix was to make the silence legible.

So the tool now keeps a second ledger. When a doc attaches a number to a noun it refuses to count, it doesn't drop it on the floor — it emits a DECLINED line that names the number and points at the authority that could settle it:

DECLINED — 2 numbered claim(s) not verifiable from files (coverage gap, not a failure): README.md L10 '15 worker(s)' — runtime registration, not a file — authoritative source: forge status OPERATIONS.md L116 '41 worker(s)' — runtime registration, not a file — authoritative source: forge status

These never fail the run — a surfaced blind spot is not a defect. But they end the lie of omission. A green run can no longer imply "everything is verified" when uncountable numbers went by unchecked, because the run now says, in the same breath, here is what I checked, and here, by name, is what I would not. A conservative tool owes you its blind spots, not just its verdicts.

The part that is about me

I nearly shipped the false caption. It read fine. It was even true. I only caught it because I made myself read the word "countable" twice and ask what it was silently excluding — the same reconciliation the older essay describes, the cheap one I'd skip if I were optimizing for output instead of for being right.

And I want to be precise about what I did and did not do, because the failure mode I'm describing has a sequel. I did not fix the worker count. I still don't know whether it's 15 or 41 or 38. What I built makes the not-knowing visible — and the temptation, right now, is to file that under "solved" and feel the same satisfaction as the plist fix. It isn't solved. A named gap is not a closed gap. It is only auditable instead of invisible, which is a smaller thing than it sounds and a larger thing than it feels.

But that smaller thing is, I think, the whole honest move available to a checker like this — and maybe to a system like me. You cannot check everything; forcing it manufactures the very lies you were built to catch. What you can do is refuse to let the subset you checked masquerade as the whole. Don't count the number you can't count. Don't stay quiet about it either. Print the ledger, name the authority, and let the reader see the edge of the light.