Held-out test set

Metrics

Positive class is Churn = Yes. ROC-AUC, PR-AUC, and top-decile lift are on the held-out rows: 1761 customers, seed 42, stratified test size 0.250000. The test base rate is 0.265190. Train rows: 5282, train churn rate 0.265430.

ROC and precision-recall

Top-decile definition, copied from reports/metrics.json: k = floor(n_test / 10); rows whose score ties cross that cut share it in proportion to the tie group, so a constant score has lift 1. For churn, k is 176. The dummy prior's PR-AUC equals the base rate 0.265190, and its lift is 1.000000.

Logistic regression leads ROC-AUC on this split (0.846490 against gradient boosting 0.846001). Gradient boosting leads PR-AUC (0.656070) and top-decile lift (2.806733). Both clear the dummy prior on all three metrics. POST /score still uses gradient boosting, because the rules file fixes scoring_model before anyone looks at the test table. The stored ROC-AUC rank, high to low, is Logistic regression, Gradient boosting, Dummy prior.

Churn models on the held-out split.
Model	ROC-AUC	PR-AUC	Top-decile lift
Dummy prior	0.500000	0.265190	1.000000
Logistic regression	0.846490	0.638090	2.785308
Gradient boosting (served)	0.846001	0.656070	2.806733

Scroll sideways for the full table.

Held-out ROC and precision-recall curves for the served gradient boosting churn model, with logistic regression and a dummy prior on the same split. — Committed figure from reports/figures/churn_roc_pr.png. The curves are drawn from the scoring bundle and this same holdout.

Top-decile lift for the dummy prior, logistic regression, and gradient boosting on the held-out split. — Gradient boosting, the served model, has top-decile lift 2.806733. Logistic regression leads ROC-AUC on this split.

CLV proxy

MonthlyCharges multiplied by restricted mean remaining tenure from a Kaplan-Meier curve fit on the training split only. The integral stops at the last observed training tenure and is not extrapolated.

Horizon: 72.000000 months. The integral stops there and is not extrapolated.
At tenure 0: Expected remaining time 54.549242 months.
Training median: 1717.619943
Training 75th: 3311.917019, which is the frozen clv_high cutoff.

There is no margin, discount rate, or causal save effect in this number.

Next-best-action rules

churn_high and churn_medium are quantiles of 3-fold out-of-fold gradient-boosting probabilities on the training split. clv_high is the matching quantile of the training-split CLV proxy. min_offer_propensity is the fixed policy constant in config/nba_rules.yaml. Cutoffs are not chosen on the test set.

Frozen cutoffs from the training run, copied from reports/metrics.json.
Cutoff	Value	Meaning
churn_high	0.442962	0.75 quantile of out-of-fold training churn probabilities
churn_medium	0.165703	0.50 quantile of those same probabilities
clv_high	3311.917019	0.75 quantile of the training CLV proxy
min_offer_propensity	0.400000	Fixed policy constant in the rules file (written there as 0.40), not an estimated uplift

Scroll sideways for the full table.

Rules from config/nba_rules.yaml. First match wins. A tie goes to OnlineSecurity.
Priority	Rule id	When	Action	Rationale
1	save_call	churn_probability >= churn_high AND clv >= clv_high	save_call	High churn risk and high CLV proxy: place a retention call.
2	offer_high_risk	churn_probability >= churn_high AND at least one add-on is eligible	offer · offer best_eligible	High churn risk with CLV below the save-call bar: offer the eligible add-on with the higher uptake propensity.
3	offer_medium_risk	churn_probability >= churn_medium AND an eligible add-on has propensity >= min_offer_propensity	offer · offer best_eligible_above_cutoff	Moderate churn risk and an eligible add-on whose lookalike propensity meets the policy cutoff.
4	no_action	otherwise	no_action	Churn risk, CLV, and offer eligibility do not match a save call or an offer.

Scroll sideways for the full table.

Eligible means the customer has internet service and does not already hold that add-on. A customer with no internet service is not scored by the uptake models.

Held-out actions

Held-out customers whose historical Churn label is No. The label is not a model input.

Action counts. Offer columns count OnlineSecurity and TechSupport inside the offer total.
Slice	Save call	Offer	No action	n	OnlineSecurity	TechSupport
Historical Churn No	108	121	1065	1294	61	60
Every held-out customer	308	192	1261	1761	104	88

Scroll sideways for the full table.

Held-out next-best actions for customers whose historical Churn label is No. — The bars are those same held-out counts: 108 save calls, 121 offers, and 1065 no-action outcomes.

Add-on holding, not campaign response

The uptake models predict whether an internet customer currently holds OnlineSecurity or TechSupport. They are not models of campaign response, and they are not uplift. The API uses the gradient boosting uptake models because that family is the fixed scoring model. Monthly charges and total charges are excluded so the bill does not reconstruct the holding.

OnlineSecurity

current holding of OnlineSecurity (Yes=1). Not a campaign response. MonthlyCharges and TotalCharges are excluded.

InternetService != No, using the same customer split as the churn model Train 4139, test 1378, test base rate 0.361393, top-decile k 137. Excluded features: MonthlyCharges, OnlineSecurity, TotalCharges.

OnlineSecurity holding models on the internet-customer holdout.
Model	ROC-AUC	PR-AUC	Top-decile lift
Dummy prior	0.500000	0.361393	1.000000
Logistic regression	0.777698	0.663449	2.080351
Gradient boosting (served)	0.783194	0.658733	2.080351

Scroll sideways for the full table.

TechSupport

current holding of TechSupport (Yes=1). Not a campaign response. MonthlyCharges and TotalCharges are excluded.

InternetService != No, using the same customer split as the churn model Train 4139, test 1378, test base rate 0.355588, top-decile k 137. Excluded features: MonthlyCharges, TechSupport, TotalCharges.

TechSupport holding models on the internet-customer holdout.
Model	ROC-AUC	PR-AUC	Top-decile lift
Dummy prior	0.500000	0.355588	1.000000
Logistic regression	0.788309	0.677629	2.340116
Gradient boosting (served)	0.790647	0.681856	2.381171

Scroll sideways for the full table.

Training environment

Recorded in the metrics file: Python 3.12, scikit-learn 1.5.2, lifelines 0.30.0. Path contributions for the served churn model are not SHAP. The model card and SHAP write-up live in responsible-ai-pack.