A wall of text is not an answer
Retrieval scores are easy to game: return more, and it looks like a hit. Curator grades its recall the way a user actually judges it, so a confident wrong answer scores worse than an honest blank, and a paragraph of near misses does not count as getting it right.
There is a cheap trick that makes almost any memory system look good on a scorecard. Return more. Hand back a fat block of loosely related text, and a lot of scoring methods will mark it correct, because the right words are in there somewhere. Your agent then reads the whole blob, picks the wrong line out of it, and acts on that.
Curator refuses to grade itself that way, because you do not experience recall as a scorecard. You experience it as whether the one thing you needed showed up where you would actually see it.
The right answer has to land where it will be read
So that is what we measure. Not whether the needed memory exists somewhere in a long list, but whether it lands in the handful of results an agent will actually read before it acts. A memory buried at position forty is not recalled in any way that matters. In production it is a miss, so in the benchmark it is scored as one.
A wall of text gets the same treatment. If the answer is technically present, folded inside a big low-relevance dump, that is not a hit. It is the system passing the work of finding the answer back to you and calling it done. We grade it as the miss it is in practice.
Confident and wrong is the worst outcome, so it scores worst
The failure that costs you the most is not the empty answer. It is the fluent, confident answer built on something that should not have surfaced: the stale note, the superseded decision, the plausible neighbor that reads close to your question and is wrong. An empty answer makes you go look. A confident wrong one makes you act.
So our grading is deliberately lopsided. Saying nothing when there is nothing solid to say is treated as a good outcome, not a blank to be penalized. A confident answer that turns out wrong is penalized harder than the honest blank. The bar we hold recall to is: say what you know, say plainly when you only half know it, and say nothing when you know nothing. A memory that can admit the last one is a memory you can leave in the loop.
Why the grading is the product
Any team can tune a memory layer to win a retrieval metric. The metric goes up and the experience does not, because the metric was rewarding volume and coincidence. Grading recall the way a user judges it, on whether the right thing lands and on whether it refuses to bluff, is what keeps the number and the felt quality pointed the same direction. The honesty is not a nice feature bolted on top of recall. It is what makes the recall worth trusting.
Where it stands
Curator is a working eval-gated pre-release product and runs on your own machine. This is the recall half of a discipline whose write half is covered in good memory knows when to refuse, and it is why feeding it more does not make it dumber, the case laid out in why AI memory gets worse. It answers to the same proof standard as BenchProctor and pairs with TheAuditor and Warden. Follow release preparation.
Was this useful?