One Model, One Shot at Whatever Blind Spot It Has Today
Ask one model a hard question and you get one shot at the truth, plus whatever bias that particular model happens to carry. The multiagent debate idea (Du et al., 2023) is the fix humans already figured out: a jury usually beats a single juror, not because any one juror is smarter, but because bad independent guesses rarely agree with each other while good ones tend to converge.
debatekit simulates that dynamic without calling a real model. Each agent is a noisy classifier with its own accuracy. Debate is rounds of exposure to the group's current answers, followed by revision weighted toward a clear plurality, then a final vote.
The Persuasion Knob Is the Interesting Part
Turn persuasion to 1.0 and the panel converges instantly to whatever the first-round plurality happened to be, groupthink with extra steps, locking in a wrong answer exactly as fast as a right one. Turn it to 0.0 and debate does nothing, it is just an expensive way to run a vote. The default of 0.6 models agents that take the group seriously without being pushovers, closer to how a real debate protocol, or a decent jury, actually behaves.
The Panel Effect Is Huge. The Question Is Whether the Extra Point Is Real
At a deliberately mediocre 55% individual accuracy, a lone agent gets there 57% of the time. A panel of five with zero communication, just independent votes, already jumps to 79 percent, the classic wisdom-of-crowds effect, and that jump is large and completely reliable. Letting the panel see each other's answers and revise for two rounds appears to buy one more point on top, landing at 80 percent. I wanted to know if that extra point was real or just where the dice happened to land.
Twenty Trials Was Never Enough to Trust That One Point
The benchmark averages 20 trials per question, framed as enough to make the numbers stable. I tested that framing directly: I resampled the exact same 20-trial sample size across ten different, equally valid, non-overlapping windows at the same 55% accuracy, instead of the single window the published number happens to come from. The deltas: plus 1.2, plus 0.8, minus 0.4, zero, plus 0.4, minus 0.4, plus 1.2, zero, minus 3.8, minus 0.8 points. The published plus 1.3 was one draw out of a spread that swings by more than five points and lands in negative territory almost as often as positive. That is not a stable measurement. That is noise wearing a headline.
So I ran it properly: 2000 trials per accuracy level, checked against 12,000-trial runs across the 0.35 to 0.60 accuracy range for good measure, with standard errors reported alongside every delta. At every single accuracy level tested, the gap between a plain independent vote and two rounds of debate never once clears one standard error. The honest reading is that in this simulation, letting the panel revise for two rounds after voting does not measurably beat voting alone. The real, load-bearing effect is the panel itself, one agent to five independent votes, a 20-plus point jump that is five to ten times the size of the noise floor. The extra debate rounds are not the free additional accuracy the original headline implied.
This is not a code bug. debate.py and agents.py do exactly what they say on the page. It is a sample-size problem in how the original number was measured, so the fix is a properly powered companion measurement rather than a change to the simulation itself. The original benchmark output is untouched and still reproduces the exact published table.
19 tests pass in CI across Python 3.9, 3.11, and 3.13.