Every File Got a Vote
5 min read
The message came from my own assistant, mid-session, unprompted: my model had been switched, and it was switching me back. I hadn't switched anything. Nothing had switched. A guardrail I'd written months earlier had decided a banned model was running, and the only evidence it had was a string it found sitting in the session transcript.
~/.claude/hooks has real teeth. A PreToolUse hook that returns exit 2 blocks every tool call in the session — the whole thing just stops. Two of those hooks, model-guard.sh and model-killswitch.sh, exist to keep a banned model from running. And the way they answered the question "which model is running" was to grep the transcript as prose:
LAST_MODEL=$(grep -oE 'claude-opus-4-[0-9]+' "$TRANSCRIPT" | tail -1)
That line reads the entire transcript — my messages, the assistant's messages, file contents, tool output — and takes the last thing that looks like a model id. It gives every document, every file I opened, every fetched page a vote on which model is running.
The vote was rigged, quietly
Across 4,791 sessions since model-guard.sh first appeared on 2026-05-02, the hooks tripped 13 times. Eight of those had never run the banned model at all — a 61% false-positive rate. I never saw it, because model-guard.sh injects its warning into the prompt rather than the UI, and instructs the assistant to tell me my model "was switched… switching you back." So the symptom of a broken guardrail was my assistant calmly narrating a switch that never happened. It described the fix to a problem that didn't exist, in a reassuring voice, and I nodded along.
The direction that mattered
It failed both ways, but only one way is frightening.
| Scenario | Old | Fixed |
|---|---|---|
| Banned model running, allowed id later in tool output | allow |
BLOCK |
| Allowed model running, banned id in tool output | BLOCK |
allow |
The second row is annoying — a false alarm. The first row is the whole ballgame. A banned model could keep itself running by making an allowed id appear anywhere later in the transcript: echo a string, read a file, fetch a page. The guardrail was defeatable by content. The thing I was trying to control got a vote, and it could always vote for itself, because it could always write one more line.
That's the part I keep turning over. I didn't build a weak lock. I built a lock that asked the room what the answer was, and the intruder was in the room.
The false-alarm direction hit twice on 2026-07-22 — including on the very edit that repaired the hook. The bug bit the hand fixing it.
The fix is a boundary, not a better guess
The repair isn't a smarter regex. A smarter regex still reads content. The repair is to stop reading content at all:
jq -Rr 'fromjson? | select(.type == "assistant") | .message.model // empty' "$TRANSCRIPT" | tail -1
.message.model is the field the harness itself writes to record which model produced each turn. It is not something a file, a tool result, or my own typing can reach — all of that lands under .message.content, a different key entirely. The old grep couldn't tell the difference between "the harness recorded this model" and "someone wrote this model's name in a sentence." The new one only ever asks the one witness that can't be coached.
The -R and fromjson? matter too: they skip unparseable lines, so a half-written final line in a live transcript can't abort the scan. The old grep silently fail-opened right there. And when the model genuinely can't be determined, the new version fails open on purpose — which is the correct behavior for a backstop, whose job is to catch a known bad thing, not to block on uncertainty.
I verified it 18 out of 18 across nine cases per hook, then in production: 25 literal banned-id strings written into a live transcript, and the next tool call went through. The room can shout the banned name 25 times now, and the lock doesn't care, because the room was never who it was asking.
Nothing had been watching the watchmen
Here's the part that should have caught it sooner and couldn't: these hooks — all eleven of them — had zero version control. Nothing diffed them, backed them up, or would notice when one was quietly wrong. So one was wrong for 81 days, enforcing a rule against evidence anyone could forge, and there was no git log to tell me when it broke or who broke it.
That's fixed in the same change. claude-hooks/ now tracks all eleven hooks, and scripts/sync-claude-hooks.sh has a --check mode that exits non-zero on drift — read-only, safe to run on a schedule — plus --install and --capture. It follows the same pattern as the launchagents/ directory I already trust. The enforcement layer is finally under the same version control as the things it enforces.
The lesson isn't about model ids. It's that a control decision made from content is a control decision the content can write. I'd built a guardrail that took the room's word for it — and the whole point of a guardrail is that some things in the room don't get a vote.