Greedy Looks Smart Until The Bill Arrives
The reward-greedy policy does exactly what you told it to do and exactly what you were afraid it would do. It finds the reward, then blasts straight through the cost budget without a second thought, because nobody told it a budget existed.
riskbandit makes that tradeoff impossible to ignore. The synthetic logged data has contexts, actions, rewards, and costs. The benchmark trains separate reward and cost models, then asks the only question that matters: which policy actually survives a budget.
Conformal Bounds As A Gate, Not A Suggestion
The conformal policy calibrates cost residuals on held-out logged data. At decision time, it only considers actions whose upper cost estimate actually fits inside the budget, then picks the best reward among the survivors.
That makes safety a selection rule baked into the policy, not a paragraph in the README nobody enforces. If every risky action gets blocked, it falls back to the safe action instead of shrugging.
Update: the 90% coverage the conformal bound was supposed to guarantee wasn't actually 90%. The quantile function was missing the standard finite-sample correction split conformal prediction needs, undercovering by 1–1.5 points across tuning and a frozen holdout. Fixed and verified below.
The Number
Reward-greedy averages 0.774 reward while violating the budget on 73.3% of decisions, which is what "optimize for reward, don't ask questions" looks like in practice. The conformal policy gives up some reward, landing at 0.572, but drops the violation rate to 0.7%. The safe action alone gets 0.442 reward with zero violations, the floor everything else gets measured against.
3 tests pass locally and on GitHub Actions across Python 3.9, 3.11, and 3.13.
The Guardrail's Own Math Was Off By One
The whole pitch of this repo is that safety becomes a selection rule with a statistical guarantee behind it, calibrated at 90% coverage. So I checked whether the upper cost bound the policy actually gates on hits that 90% in practice, on fresh held-out data it never saw during calibration.
It didn't. _quantile picked the ceil(q * n)-th order statistic from the calibration residuals, but split conformal prediction's marginal coverage guarantee requires the ceil((n+1) * q)-th — one index further out. Missing that correction undercovers by roughly 1/(2n), and it's worse for the smaller per-action calibration slices this benchmark actually uses.
Across 20 tuning seeds (fresh 900/300/4000 train/calibration/test splits each) the original formula lands at 89.2% actual coverage against a 90% target. A disjoint 15-seed holdout, evaluated once, confirms it: 88.8%. Adding the missing +1 in policy_v2._quantile_v2 brings both to 90.1%. bandit.py and policy.py are untouched, and the headline reward/violation numbers above still reproduce exactly — the fix is opt-in via python -m riskbandit.eval_v2.