Clairmont Supply
Contact

Working note 03, , Mahé Chadeffaud and Hugo Marotin

Out-of-sample evaluation of the buyer selection model

Abstract

The selection model estimates, before a bid, the probability that the buyer pays on time. We use it only if it predicts better than a simple rule, the buyer's own past payment record, on contracts it has never seen. We judge it with the Brier score, check its calibration, compare it with placebo models trained on shuffled outcomes, and repeat the comparison period by period in a walk-forward design. All figures use simulated contracts with a fixed seed.

1. The question and the score

For each contract the model gives p_i, the probability of payment within 30 days of the due date, and the outcome y_i is 1 if that happened. The Brier score is the mean squared error of these probabilities, and the skill compares it with the base rate b of on-time payment:

B = (1/n) Σ (p_i − y_i)² skill = 1 − B / B_base, B_base = b·(1 − b)

A skill of zero means the model does no better than quoting the base rate to every buyer. The model only sees what was known on the day of the tender: the buyer's history up to that day, its type and region, the goods, the amount and the payment terms.

2. Calibration on unseen contracts

The model is trained on older contracts and judged on later ones. We group its predictions in ten bins and compare the mean prediction of each bin with the observed rate.

SELECTION MODEL CALIBRATIONCalibration · 4,812 unseen contractsselection modelbuyer's past recordperfect calibration00.250.50.75100.250.50.751predicted P(paid on time)observed rateBrier 0.163 · past record 0.192 · base 0.20910 bins · trained on earlier contracts · simulated, not live results
Figure 1. Calibration of the selection model and of the buyer's past record on unseen contracts.

3. Placebo

A model can look skilful because of how it was built rather than what it learned. We retrain it 300 times on outcomes shuffled at random, which removes any real relation, and score each placebo on the same unseen contracts.

PLACEBO TESTBrier skill · model vs 300 placebosshuffled-label placebosbuyer's past recordselection model00.10.20.300.51Brier skill vs base raterelative frequencyskill 0.223 · placebos max 0.001 · 0 of 300 aboveplacebo = same model trained on shuffled outcomes · simulated, not live results
Figure 2. Brier skill of 300 shuffled-label placebos, of the buyer's past record and of the selection model.

The placebos sit around zero. A model whose skill fell inside their range would be discarded, whatever its score.

4. Walk-forward

Payment behaviour changes over time, and so do procurement rules. We split the history into consecutive periods, train on everything before each period, and test on the period itself.

WALK-FORWARD EVALUATIONBrier skill by period · 8 periodsselection modelbuyer's past recordplacebo12345678-0.100.10.20.3test period (walk-forward)Brier skill vs base ratemodel beats past record in 8 of 8 periods700 contracts per period · simulated, not live results
Figure 3. Brier skill by test period for the selection model, the buyer's past record and a placebo.

The decision rule is written before the test: the model replaces the past-record rule only if it beats it in most periods and on the pooled unseen contracts. Otherwise the simple rule stays.