Your "Clarity" Score Is Quietly Punishing Good Answers
Here is how every eval rubric gets born. Someone writes five criteria in a meeting, everyone nods, and it becomes scripture. Nobody ever goes back to check whether those five criteria predict anything at all.
So I checked. In this project's own benchmark, "clarity" was not just useless, it was anti-correlated with success. The rubric was actively docking points for being clear. Somewhere out there, your rubric has a "clarity" too, and you have no idea which one it is.
rubricagent treats the rubric as something you learn, not something you defend forever. Score your traces, correlate each criterion against what actually happened, then reweight, prune, and grow it into an honest proxy for skill instead of a vibe check with a spreadsheet attached.
Three Moves Per Pass
• Reweight: each criterion's weight becomes its actual correlation with success, not a number someone guessed in a doc.
• Prune: the dead weight gets cut, and the criteria secretly working against you get cut first.
• Discover: it mints new criteria out of the words that show up constantly in wins and rarely in losses, the signals nobody thought to write down.
• Bring your own judge: the offline grader keeps CI honest and reproducible, and a real LLM grader drops in through the same interface for production.
The Demo That Refused to Flatter Me
The hardest part was making the example honest instead of impressive. An early version showed a cozy 78% on a deliberately half-baked response, which was not what the code actually did. Running it for real spat out 14% with completely different reasoning. I shipped the 14%. A demo that lies about your tool is worse than no demo.
AUC as the quality metric was also on purpose. It measures how cleanly the rubric separates real wins from real losses, so "better rubric" has a definition you can check instead of an opinion you can argue about.
The Number
Start with a flat rubric where only one of five criteria actually predicts anything. One evolution pass drags rubric quality from 0.774 to a perfect 1.000, pruning the liars and surfacing the real signal on the way.
The Grounding Criterion Was Built to Grade Its Own Test
I went looking at how the benchmark actually generates its "grounding" examples. It concatenates a marker string: "according to the source the tool search found evidence". The grounding criterion's keyword list is ["source", "cite", "according", "tool", "search", "found"]. Five of six words appear verbatim in the sentence that decides the label. The 0.774-to-1.000 climb I'd been quoting measures whether keyword-overlap scoring can detect data built from keyword overlap. It measures nothing about whether the grader can read a transcript.
So I ran the same grader against ordinary language: "The SEC 10-K filing lists a March launch" versus "I think it probably launches next year." The grounding criterion's correlation with the true label was 0.000. Not weak, blind. It scored every genuinely grounded trace and every genuinely ungrounded one at exactly zero, because none of them happened to contain one of six exact words.
What made this hard to catch is that AUC after evolution still climbed to 0.944 on the same corpus, which would have looked perfectly fine reported on its own. The reason: this project's separate "discover new criterion" mechanism mines raw trace text directly and doesn't touch the grader at all. It quietly covered for a criterion that had never worked, and nothing in the benchmark's own printed output would have told me.
Two Bugs I Only Found Building the Fix
RubricEvolver.evolve() constructed Judge(rubric) with no grader argument, twice, regardless of what you'd passed to RubricEvolver(grader=...). There was no way to actually evolve a rubric against a better grader, or a real LLM, only to call judge() once with it and watch evolution silently fall back to the default anyway.
And the "discover new criterion" step judged significance with a bare proportion gap and no minimum sample size. On text where the label was a pure coin flip, unrelated to a single word in it, that produced a confidently-reported, entirely fabricated discovered_signal in about 1 run in 25. Requiring the term to actually appear in a real share of successes, not just clear a small gap that a handful of coincidental hits satisfies, brought that to 0 in 200 trials without weakening the real discovery this benchmark depends on.
The fix widens each criterion's evidence vocabulary from a handful of words to what citation, correctness, completeness, and clarity actually look like in text, "filing," "disclosure," "confirmed," "per," plus basic stemming so "cited" matches "cite." Held out and evaluated exactly once, after the vocabulary was frozen: correlation 0.447 to 0.834. 21 tests now, and CI fails the build if grounding correlation regresses on realistic text.
I did not silently swap the default grader. Doing that would have changed the published 0.774 baseline this README quotes, since the wider vocabulary also fires more often on the synthetic benchmark's own filler text. The fixed grader is available explicitly instead of changing a number I'd already told people to expect.