Your Churn Model Is an Ex Who Thinks You're Still Together
Most churn models get trained once, shipped, and then ghosted. Then a price change lands, or a competitor shows up, or the product pivots, and the whole reason customers leave quietly rewrites itself. The model has no clue. It keeps answering with total confidence based on a world that stopped existing months ago.
And the worst part is how slow you find out. Accuracy does not crash. It erodes. By the time a dashboard looks wrong enough for someone to notice, you have been making bad calls for three weeks.
churnfm watches the shape of its own predictions with the Population Stability Index and retrains the second that shape shifts, instead of waiting for a human to feel like something is off.
A Loop, Not a Model
The classifier was never the interesting part. The loop wrapped around it is.
• Drift detection: it compares the distribution of predictions now against a reference window, and catches the relationship shifting before accuracy visibly falls off a cliff.
• Auto retraining: the moment PSI crosses the line, it retrains on recent data and resets its baseline.
• Recent window, not everything: it retrains on a sliding window on purpose. Dump in all your old data and you drag the dead relationship along with you.
• Bring your own model: the default is a stdlib logistic regression, but anything with fit and predict_proba drops into the same monitor.
The Instinct That Was Wrong
My gut said retrain on all the history. More data, safer model, right? Wrong. Mixing the old world and the new one keeps the drift alarm screaming forever, and the model never settles into what is actually true now. The recent-window fix looks obvious written down. It only became obvious after I benchmarked both and watched the naive version thrash.
I also scored it on precision-at-k instead of a fixed cutoff, because churn is a single-digit-percent event and a fixed probability threshold is basically noise when the thing you care about is that rare.
The Number
On a subscription stream with a drift injected halfway through (price-sensitivity suddenly becomes the thing that matters), the static model stalls at 80% post-drift precision. The version that notices and retrains itself climbs back to 89%.
I Went Looking for the 9 Points, and Found a Confound Instead
Something about that 80%-to-89% story bothered me: the static model's post-drift precision was 80%. If the model's understanding of the world had genuinely gone stale, it shouldn't be scoring 80% on anything. So I checked what the injected "drift" actually did to the data, and it changes three things at once at the midpoint: the price-sensitivity coefficient, the distribution of price increases, and, as a side effect of both, the churn base rate. Which jumps from 3.5% to 69.5%.
A 69% base rate makes churn nearly a coin a model can call correctly just by leaning toward "churned." That's most of why even the never-retrained model scores 80%. The benchmark's headline number wasn't really testing whether PSI catches drift. It was testing a scenario that happened to get easier at the exact point drift was injected.
Isolate the Relationship Change, and PSI Never Fires
So I built a version where the base rate and every input distribution are held constant by construction, mathematically centered so the swap in coefficients doesn't itself shift the average predicted probability, and only the relationship between features and outcome inverts. Engaged, long-tenure customers were safe before; they're the ones leaving now. Same inputs, opposite meaning.
PSI never fires. Zero retrains, for the rest of the stream's life. Measured directly: PSI sat under 0.05 against a threshold of 0.25 while precision fell from 42% to 14%. The reason is almost embarrassingly simple once you see it: PSI compares the distribution of predicted scores, and a logistic model re-scores the same input distribution through the same fitted function whether the relationship underneath has changed or not. The scores look exactly as healthy after the world flipped as they did before. PSI was watching the thermometer while the room changed temperature scale.
What Actually Catches It
PSI compares scores to scores. The fix compares the model's predictive quality on labeled outcomes, reference window against current batch, using a log-loss ratio. A model whose relationship to the world inverted gets measurably worse at labels it's never adjusted for, even when its score distribution hasn't moved an inch. The new monitor retrains on either PSI or this outcome signal firing, and on the isolated scenario, outcome drift is the one doing all the work.
The trade is honest: PSI can flag drift before any label exists for new data. Outcome drift needs labels, which in a real churn pipeline arrive weeks after the fact. Run both, for different reasons.
The Fix That Made It Worse First
My first attempt kept the existing sliding retrain window, the last two batches of labeled history, and just wired in the new outcome trigger. Precision after retraining stayed at 9 to 12%. No better than never retraining at all.
The window was mixing labeled examples from both regimes: half teaching the old relationship, half the inverted one. A model fit on that mixture learns something close to nothing, confidently. Retraining on only the single batch that tripped the alarm, guaranteed to be from the current regime, recovered precision to 38 to 51%. Under real concept drift, a wider retrain window isn't more data, it's contradictory data.
Held out and run once, on a differently-shaped drift written after the outcome threshold was frozen (tenure and support tickets swap which one protects against churn, instead of usage and tenure): PSI still never fires. The dual-signal monitor recovers from 21% to 45% with one retrain, triggered entirely by the outcome signal. 30 tests now, and CI fails the build if the dual-signal monitor ever stops catching what PSI misses.