The Average Client Is A Fiction Everyone Agreed To
FedAvg is easy to explain because every client politely disappears into one neat average. Real clients do not disappear. They carry different feature shifts, different label balances, and different, creative ways to punish a global model that was never built for them specifically.
fedcal builds that mess directly into the benchmark. Six clients get genuinely different feature shifts and class skews, then the repo compares plain FedAvg, FedProx, and a conservative local calibration step against each other on the client nobody wants to be.
Personalization With A Seatbelt On
The local calibration step only moves a client's bias if validation accuracy actually improves. That matters because personalization overfits fast the moment the local slice of data gets small, which is most of the time in the real world.
Client updates are clipped, the logistic model is readable end to end, and the whole benchmark stays dependency-free. This is not a full federated platform pretending to be production infrastructure. It is the failure mode, isolated and put under a microscope.
The Number
FedAvg gets 0.829 macro accuracy while stranding its worst client at 0.667, which is exactly how a model quietly fails one real user while the dashboard reads green. FedProx plus conservative calibration trades 1.4 points of macro accuracy for a 2.7 point gain on that worst client, landing at 0.694, on purpose.
The Seatbelt Doesn't Actually Buckle
I said above that the calibration step only moves a client's bias if validation accuracy actually improves. That is true, mechanically. What I had not checked was whether validation accuracy on a thirty six point slice actually predicts test accuracy on a different thirty six point slice from the same client. It does not, reliably.
I ran the same benchmark across sixty different seeds instead of the one I had been quoting. The mean gain on the worst client from calibration is negative, down 2.1 points on average, and it makes the worst client's accuracy worse more than twice as often as it makes it better, twenty eight losing seeds against thirteen winning ones. Seed twenty one, the number sitting in the README above, was one of the thirteen lucky draws.
I tried the obvious fix. Calibrate on the combined train and validation data instead of validation alone, three times as many points to estimate an offset from, and only apply that offset if it clears a real improvement margin instead of any positive fraction of a percent. Tuned against thirty seeds it looked like a genuine fix, a positive average gain, ten wins against three losses. Then I built a separate set of twenty seeds I had not touched during tuning and evaluated the fix against it exactly once. The improvement did not survive. Mean gain went negative again, three wins against eight losses.
I am reporting that failure instead of quietly discarding it, because it is the more useful result. It means the instability here is not a hyperparameter you can tune your way out of. It is a sample size problem. With only six clients and thirty six to a hundred and eight points of calibration data per client, worst client accuracy moves by roughly two point eight percentage points for every single test example that flips. No offset selection rule built on that little data is going to reliably beat that noise, and pretending otherwise with a better-tuned margin would just be finding a new way to overfit the same small sample.
The original training code and the published table are both untouched, and they still reproduce exactly. 14 tests pass locally and on GitHub Actions across Python 3.9, 3.11, and 3.13.