Your Rubric's "Clarity" Criterion Might Be Penalizing Good Answers
Somebody on a team writes a five-criteria rubric for grading agent transcripts, everyone nods, and it's been treated as gospel ever since. Nobody has checked whether all five criteria actually predict anything. In rubricagent's own benchmark, "clarity" turned out to be actively anti-correlated with success. The rubric was penalizing the agent for being clear.
rubricagent treats the rubric as something you learn, not something you write once and defend forever. Score your agent traces, correlate each criterion against real outcomes, then reweight, prune, and grow the rubric so it becomes an honest proxy for capability instead of a vibe check with a scoring column.
Technical Architecture
One evolution pass does three things:
• Reweight: every criterion's weight becomes its correlation with actual success, not an arbitrary hand-assigned number.
• Prune: criteria below a signal floor get dropped, anti-signals like the "clarity" example first.
• Discover: a new criterion gets minted from evidence terms that show up constantly in wins and rarely in losses, criteria nobody thought to write down by hand.
• Bring your own judge: the deterministic heuristic grader is offline and reproducible for CI; a real LLM grader wires in through the same interface for production judging.
Challenges & Solutions
The genuinely hard part was making the demo honest instead of impressive. An earlier version of the example output showed a flattering 78% score on a deliberately thin, half-finished response, which wasn't what the code actually produced. Running it for real gave a 14% score with completely different rationale text. Fixing the README to show the real number, warts and all, mattered more than a demo that looks good but lies about what the tool does.
Choosing AUC as the quality metric was also deliberate: it measures how well the aggregate rubric score separates traces that truly succeeded from ones that failed, so "better rubric" has an actual, checkable definition instead of being a matter of opinion.
Impact & Results
Starting from a flat, equal-weight rubric where only one of five criteria actually predicts success, one evolution pass lifts rubric quality (AUC against ground truth) from 0.774 to 1.000, pruning the misleading criteria and discovering the real signal along the way.
6 tests pass in CI across Python 3.9 through 3.13, plus the example and benchmark scripts run on every push, zero API keys required.