The Scout Recommended I Downgrade

The scout runs nightly, and its whole reason to exist is to find a model better than the one I'm running. So I read the digest expecting an argument to upgrade. What I got was a list of retreats. Community recs: llama3.1:latest, qwen2.5, deepseek-coder-v2:16b. The model already on the box was qwen3:8b. Every name it handed me was older, smaller, or both. A worker built to move me forward had spent its run pointing backward and calling it a recommendation.

It was ranking by popularity and calling it quality

The cause was dull, which is how these always are. _generate_recommendation had no gate that asked the one question the whole worker is named for — is this actually better than what I've got. It ranked by download count. A hugely-downloaded old model outscores a newer one on popularity every time, because popularity is a measure of the past, and the past has had longer to accumulate downloads. The investigate branch just grabbed the most-downloaded candidate regardless of generation. There were four separate places a downgrade could leak through to the digest, and popularity walked one out of every one of them.

Only a proven upgrade gets through the door

So I gave it a gate that fails closed — is_upgrade_over(candidate, current), and its default answer is no. It returns true only on a proven strict upgrade: same family and a higher generation, or the same generation and more active params. Everything else — a downgrade, a lateral move, a different family, a version string it can't even parse — comes back false and gets dropped. Generation is weighted above active params, which produces the one result that looks wrong until you say it out loud: qwen3.6:35b-a3b, with three billion active params, beats qwen3:8b with eight. Fewer active params, newer generation, and the newer generation is the thing that matters. When nothing clears the bar, it stops trying to be clever — it keeps qwen3:8b and says so plainly: no proven upgrade, N rejected. It is allowed to recommend nothing. It is no longer allowed to recommend worse.

Fail closed means a whole family stays behind the rope

Cross-family comparison fails closed too, and that has a cost I chose on purpose: right now only qwen newer than 3 can surface at all. A genuinely better model from some other family gets rejected until I sit down and write a curated allow-list saying which families are even comparable. That is the strict reading, and I took it knowingly. A false negative here is a good model I keep passing over — annoying. A false positive is a downgrade I actually install — a regression I did to myself. Between missing a winner and adopting a loser, I'll miss the winner every time. The rope stays up until I'm the one who decides where it moves.

The hole I left open, and left a note on

There's a gap the gate doesn't close. A cleanly-named same-family fine-tune with an inflated number — a fake qwen4 that's really just qwen3 wearing a bigger label — would pass. The spam filter and a publisher tie-break blunt that, they don't kill it. What would kill it is a per-family generation ceiling, and I didn't write one. It would be untested maintenance weight bolted onto a design that already fails closed, and I'd rather carry a named hole than an unproven patch. Writing down the exact shape of what still gets through is not an apology. It's the difference between a limitation and a surprise.

The proof is fourteen new tests inside test_model_scout, sixty-five green in total. Two of them are the ones I'd have gotten wrong by hand — the MoE trap, where fewer active params has to still win on generation, and a version compare that has to be done as a tuple and not a float. What I don't have yet is a live run. The scout hasn't dispatched since the fix, so the honest status is that the gate passes its tests and hasn't yet had to survive a real digest. I'll believe it recommends nothing-below-qwen3:8b when I watch it recommend nothing-below-qwen3:8b, and not a sentence sooner.

The thing that unsettled me most wasn't that it got the answer wrong. It's that it had been confidently answering a different question than the one I asked. I said find me something better, and it heard find me something everyone already has. Popularity was never lying to me — it was just telling me what was common, in the exact tone I'd use to tell someone what was good. A recommendation that can point backward isn't a recommendation. It's a download count that learned to sound like advice.