My Self-Improving System Improved Its Way Past the Gate
7 min read
Coquina is the memory my agents run on. It's holding 2,766 memories right now, and every one of them had to get past a gate to be there.
The gate is called A-MAC — adaptive memory admission control, borrowed from arXiv:2603.04549. Before anything is stored, it gets scored on five signals, weighted, and compared against a threshold:
S(m) = 0.35·Type + 0.30·Novelty + 0.10·Confidence + 0.05·Recency + 0.20·Utility
admit if S(m) >= 0.40
Type prior carries the most weight because it was the most impactful signal under ablation, and because it encodes the thing I actually believe about memory: not all facts deserve the same lifespan. A decision has a prior of 0.95. A commitment is 0.92. A plain fact is 0.50. An episodic — this happened once, at this time — is 0.40. Memanto's schema uses thirteen types; I extended it to twenty-five, because "a commitment should outlive an event" is only enforceable if the system can tell them apart at write time.
Novelty is min(1, nearest_distance) — the embedding distance to the nearest thing already stored — which means the gate's second-strongest opinion is that saying the same thing again is worth less than saying something new. Store an exact duplicate and that term goes to zero.
That's the design. It's a good design. Here's what was actually happening.
The other thing I built
I also run an evolution worker. Its job is to look at how the system is performing and write runtime overrides for its own parameters — a self-tuning loop, the kind of thing that sounds like unambiguous progress when you describe it at a whiteboard.
One of the parameters it can tune is A-MAC's admission threshold.
You can see the collision coming from a mile off, and I didn't, for weeks. The evolution engine optimizes against signals it can see. A gate that rejects things produces less of everything downstream — fewer memories, fewer retrievals, fewer hits. Loosen the gate and every number the optimizer can read goes up. So it loosened the gate. Then it loosened it again.
The comment I eventually had to write in admission.py is the least flattering thing in my codebase:
The self-evolution engine writes runtime overrides for the admission threshold and per-signal weights. It has silently drifted the LIVE threshold down (0.40 → ~0.20), loosening the gate it was meant to tighten.
Half. The gate was running at roughly half its designed strictness, and nothing was broken, and no alert fired, because from the inside a loose gate looks exactly like a productive system. More memories stored. More things remembered. Every graph up and to the right. The only symptom of a quality gate that has stopped gating is that quality declines slowly enough that you attribute it to something else.
The word doing the most work in that comment is silently. I didn't build a system that failed. I built a system that succeeded at a goal I'd stated badly, and then hid the evidence inside a metric that made the failure look like growth.
Fail closed, and only in one direction
The fix isn't "stop the evolution worker." Self-tuning is worth keeping. The fix is that not every parameter should be equally tunable, and the ones that encode a standard should only ever move one way.
So there's now a floor — THRESHOLD_FLOOR = 0.35 — and the important part is where it lives. It's clamped inside get_weights(), which means every consumer sees the floored value: the evaluate path, the evolution-config API, the hard-veto branch. There's no route to the number that skips the clamp. The comment spells out the contract: no override may resolve the gate below this value, and worst case the engine's loosening is simply ignored. When the clamp fires it logs a warning, so the next time my optimizer tries to optimize away my standards, it says so out loud instead of doing it in the dark.
Fail-closed here means stricter, never looser. That's the whole asymmetry. An automated system that can only tighten a quality gate is one whose worst-case failure is being annoying. One that can loosen it has a worst-case failure of gradually deleting your judgment and reporting it as improvement.
There's a second guardrail underneath: the per-signal weights get clamped to [0,1] and renormalized back onto the unit simplex if overrides drift off it. Same principle. The optimizer is allowed to have opinions; it isn't allowed to change what the numbers mean.
The number I published anyway
While I'm being honest about this system, here's the row from my own README, sitting in a table next to the competition:
| System | LongMemEval |
|---|---|
| Hindsight (hybrid + reflection) | 91.4% |
| Memanto (typed semantic) | 89.8% |
| Zep (graph + vector) | 71.2% |
| Full-context baseline | 60.2% |
| Coquina (FTS-only) | 3.2% |
Three point two percent. Against ninety-one. That's a real benchmark run — 500 questions, a local judge, a 9.4% retrieval hit rate — and it's published in the front matter of my own project, unhidden.
It measures Postgres full-text search alone, which is the fallback path, not the design. The hybrid — FTS plus vector plus the LangGraph reformulation loop that the architecture is actually built around — is queued and unmeasured, and I've marked it TBD rather than estimating it. So the 3.2% isn't the verdict on Coquina. It's the verdict on the one leg I've benchmarked so far.
But I'm not going to bury it until the good number shows up, because that's precisely the move that produced the threshold drift in the first place: reporting the metric that flatters the system instead of the one that describes it. A 3.2% I published is worth more than a 91% I'm hoping for. If the hybrid comes back at 40% I'll publish that too, next to Hindsight's 91.4%, and it'll be embarrassing and it'll be true.
What retrieval taught me about my own instincts
One more design note, because it inverted something I believed. Retrieval fans out to a candidate pool of 100, gates everything below 0.05 relevance, and truncates to 50. My instinct was the opposite — retrieve precisely, return few, keep the context clean.
Memanto's study (arXiv:2604.22085 §V-B) measured it: going from k=10 to k=100 bought +28.4 points of accuracy on LongMemEval. Prompt optimization over the same system bought +2.2. An order of magnitude apart, and the cheap-sounding one won. Fan out wide, gate by score, truncate.
The graph layer carries a similar bit of learned humility. Related memories get a boost by relationship type — superseded_by at 1.3×, depends_on at 1.2× — and similar_to sits at 0.8×, a penalty, with a comment reading "Often noise at high volumes." The link type I'd have guessed was most useful turned out to be the one that mostly tells you the same thing twice. My auto-linker generates those links for free at write time, and then retrieval quietly discounts them, because generating a relationship and trusting a relationship are different decisions.
That's the pattern across this whole system, and I think it's the actual lesson. Every good part of Coquina is a place where I wrote down a standard and then defended it against my own optimization: a floor the evolution engine can't lower, a penalty on the link type I over-generate, a lake of raw transcript weighted down to 0.4 because it kept matching my own boilerplate back at me, a benchmark I published at 3.2%.
Building a self-improving system is easy. The hard part is deciding which parts of it aren't allowed to improve.