Your Churn Model Is an Ex Who Thinks You're Still Together
Most churn models get trained once, deployed, and then quietly ghosted. Nobody's watching when a pricing change, a new competitor, or a product pivot rewrites the actual relationship between your features and who leaves. The model doesn't know any of that happened. It just keeps answering questions based on a world that no longer exists.
churnfm watches the prediction distribution itself using the Population Stability Index, and retrains the moment the underlying relationship shifts, instead of waiting for a dashboard to look wrong three weeks later when someone finally notices the numbers are off.
Technical Architecture
The point was never the model architecture, it's the monitor-and-retrain loop wrapped around whatever model you plug in:
• PSI drift detection: compares the distribution of predicted probabilities in a reference window against the current window, catching a shift in the underlying relationship before accuracy visibly collapses.
• Automatic retraining: when PSI crosses a threshold, the model retrains on a recent sliding window and resets its reference distribution.
• Sliding window, not full history: retraining only on recent data, rather than all accumulated history, matters, since mixing in stale pre-drift examples would keep re-triggering the drift alarm indefinitely.
• Bring your own model: the built-in classifier is a pure-stdlib logistic regression, but any object with a fit/predict_proba interface plugs in through the same `ChurnMonitor` wrapper.
Challenges & Solutions
The interesting design decision was the sliding window for retraining. My first instinct was to retrain on everything accumulated so far, which seems safer. It isn't: mixing pre-drift and post-drift examples together keeps the PSI monitor perpetually convinced something is still shifting, and the model never actually stabilizes into the new relationship. Restricting retraining to a recent window was the fix, and it only became obvious once I benchmarked both approaches side by side.
I also chose precision-at-k over a fixed probability cutoff for evaluation, since churn is a single-digit-percent event in most real datasets and a fixed cutoff is close to meaningless when the positive class is that rare.
Impact & Results
On a synthetic B2B subscription stream with a concept drift injected at the midpoint (a pricing change makes price-sensitivity the dominant churn driver), a static model's post-drift precision stalls at 80%. The adaptive version, which catches the shift and retrains automatically, recovers to 89%.
6 tests pass in CI, along with the example script and the full drift benchmark run on every push, zero API keys required.