The Author Can't Grade Its Own Homework

The first time I let one of my homelab's workers write up a piece of work, it handed me something clean, confident, and wrong. Not wrong in a way you'd catch skimming — wrong in two specific sentences. It said a thing had happened weeks earlier. It had happened that same day. And it explained, in fluent detail, how a fix worked — when the source it was handed never said how, only that the fix had been made. The model had reached the edge of what it was given and kept talking in the same steady voice.

The plausible errors are the dangerous ones

A typo announces itself. A fabrication doesn't — it wears the same voice as the facts around it. Neither of those two sentences was hedged. There was no "roughly" or "I think," no seam where the knowing stopped and the guessing began. They were stated as flatly as the true sentences on either side, which is exactly why they were dangerous. Generated prose fails confidently and specifically, and confident, specific, wrong is the hardest kind of wrong to catch.

The second error had a dull cause worth knowing. The source I'd fed the writer had been truncated to a roughly 700-character window, and the cut happened to land between a change's "problem" and its "fix." The model saw the problem, never saw the fix, and invented one to fill the gap. A hole in the source becomes a fabrication in the draft. I widened the window to 2,500 characters so a source can't end mid-story — but that only closes one hole, and the model can always find another.

Guilty until the source proves it

So the real fix wasn't a better writer. It was a second process whose entire job is to disbelieve the first. I gave it one governing instruction: assume every claim is unsupported until the source proves it. Not innocent until proven guilty — the other way around. Its job is to find unsupported claims, not to admire the prose.

It flags, without mercy: any timespan or date the source didn't state, any mechanism it didn't describe, any count or filename it doesn't contain, any characterization of a thing beyond what it actually says, any event it doesn't record. And it deliberately leaves alone everything that isn't a factual claim — opinion, metaphor, a rhetorical question, the closing line. It hunts invention, not personality. A good write-up with three real specifics beats a great one with a single invention, so the gate is tuned to protect the facts and let the voice be.

The author can't grade its own homework

The rule that makes the whole thing work is the smallest one: the checker is not the writer. It runs in a separate context that never watched the draft get written. It gets the source, it gets the text, and it gets nothing else — no memory of the sentences it was proud of, no stake in them surviving.

Ask a model to check its own output and it will pass it. Of course it will — it's the same mind that just decided those words were good, reading them again and agreeing with itself. Grading your own homework isn't grading. So the grader is a stranger to the draft, and it's rewarded for finding fault rather than for feeling finished.

Fail closed

If that checker errors, times out, or hands back something I can't parse, the verdict is not "probably fine." It's ungrounded, and ungrounded does not ship. "I couldn't verify this" and "this is false" get the exact same treatment: the text stays put. A verifier that waves things through whenever it breaks is worse than no verifier at all, because you've started to trust it.

Flagged claims get exactly one repair pass — each one corrected to the source or cut, no new claims allowed in — and then a single recheck. Not a loop. Text that can't be grounded in two tries is rejected, not sanded down until the checker gets tired and lets it by.

Trust the grep, not the opinion

The last lesson was the one I didn't see coming. The recheck would sometimes re-flag a claim the repair had already fixed — insisting the text still said the wrong thing when it plainly no longer did. The model wasn't reading the text in front of it; it was re-reading its own earlier complaint and restating it from memory.

So the final step isn't a model at all. Before any surviving complaint is allowed to kill a draft, I grep the actual file for the exact words it's objecting to. If the quote isn't there, the objection is stale, and the grep overrules it on the spot. The model gets to raise objections. Only the file gets to confirm them.

None of this makes the writer honest. It makes honesty the only thing that survives contact with the gate. A model will always tell you your draft is clean — fluent, confident, done. The discipline is to never let it be the one that decides. The writer gets a voice. The source gets the last word.