Benchmarks
Benchmark results
Each scenario uses synthetic data with a known ground truth, so results can be checked against the correct answer. Values are means ± standard deviation over 5 runs. The full suite runs in 196 seconds on a laptop using 4 cores (arm64, Python 3.10.10).
| Scenario | Result | Interpretation |
|---|---|---|
| Customer retention Two businesses, 500 customers with irregular purchase intervals; two thirds of churners slow down before leaving | AUC 0.976 ± 0.005 vs 0.969 ± 0.005; Brier 0.043 ± 0.004 vs 0.061 ± 0.006 | Ranking comparable to the recency rule; probability estimates about 30% more accurate (Brier score) |
| Equipment failure 240 assets; each skipped maintenance in the last 60 days triples failure probability | AUC 0.674 ± 0.050 baseline, 0.735 ± 0.059 generic, 0.767 ± 0.043 with a plugin | The generic task detects the signal; one domain feature improves it further |
| Length of stay 300 patients; stay length increases with recent admissions | MAE 1.61 ± 0.06 days vs 2.18 ± 0.14 for the previous value | Value prediction outperforms the previous-value baseline |
| Random data 300 entities with a memoryless 4% weekly event rate | Models promoted: 2 of 25 with the default gate; 11 of 25 with the confidence check off; promoted models worse than the baseline: 0 | The default gate requires the gain to be clear at 90% confidence and keeps the baseline on most random datasets. Without that check, a model that only ties the baseline passes about half the time |
| Intervention uplift 800 customers; the intervention raises response by 30 points for overdue customers only; 25% held out | Top-ranked half 23.6 pts ± 5.9; bottom half -1.8 pts ± 5.2 | Uplift ranking separates responsive from non-responsive customers |
| Offer effect on order value Half of interventions include an offer that reduces order value by 20% | Estimated -20.2% ± 0.9 | The order-value model recovers the true effect |
| Insight detection 600 new customers; one first-order product doubles the repeat rate; a matched dataset with no effect | Planted pattern detected 5/5; false patterns on random data: 0 | False-discovery control holds |
| Non-linear effects 400 assets; failure risk is high outside a 40–60 degree operating band on either side | AUC linear 0.670 ± 0.024, boosted 0.690 ± 0.022, auto 0.676 ± 0.024 | Boosted trees capture the U-shaped effect; auto chose them in 1/5 runs |
| Time to next event 400 customers with purchase rhythms of 5 to 40 days; half eventually stop | AUC 0.947 ± 0.014 without, 0.954 ± 0.013 with the survival feature | Better in 5/5 runs on rhythm-driven data |
| Effect of any action 1,500 machines; a programme lowers 30-day failure from 50% to 25% for old machines only; 30% held out | Estimated effect: old -0.266 ± 0.035, new -0.018 ± 0.021 (true −0.25 and 0) | The uplift model identifies which entities the action helps |
| Effect without a control group 1,000 entities; treatment is more likely where the outcome is already higher; true effect +1.0 | Matched estimate 1.017 ± 0.019; naive difference 3.076 ± 0.131 | Matching removes the confounding; the 90% interval covered the truth in 4/5 runs |
| Sequential test Baseline 10%, smallest worthwhile rate 15%, alpha 0.05, power 0.8; 600 streams per run | False alarms 0.041 ± 0.014; detection 0.799 ± 0.017; 147 observations on average against 253 for a fixed-sample test | Error rates on target with fewer observations |
| Drift Feature rows from the same distribution, and with one input shifted by one standard deviation | Largest PSI 0.011 ± 0.004 when stable, 0.931 ± 0.037 when shifted | Verdicts: stable, and retrain with the shifted input named |
| Lagged sequences 500 customers; 60% of Apparel orders are followed by Cosmetics 18 to 22 days later | Found 5/5; median lag 20 days; false patterns on random data 0.2 ± 0.4 per run | Patterns and their timing are recovered |
| Automatic tuning The 240-asset plant; the search may switch learner, add the survival feature and add any of four candidate features: skipped maintenance and three random numbers | Real feature kept 5/5; random features kept 0.0 per run; tuned model promoted 2/5; AUC 0.761 ± 0.047 tuned vs 0.735 ± 0.059 untuned | Changes are kept only when most validation folds improve; the gate promotes the tuned model only when the held-out gain is clear |
| Two live arms Arm A converts at 10%; B at 10% (no difference) or 13%; checked after every observation | False alarms 0.042 ± 0.007 against 0.363 ± 0.024 for a z-test checked every 100 pairs; detection 1.000 ± 0.000, median 1,587 per arm | Continuous monitoring without the inflated false-alarm rate of repeated checks |
| Doubly robust effect 3,000 entities; treatment more likely where the outcome is already higher; outcome curved (exp), true effect +1.0 | AIPW 0.906 ± 0.171; outcome model alone 0.849 ± 0.010; naive 2.450 ± 0.121 | Consistent although the outcome model is wrong; the 90% interval covered the truth in 5/5 runs |
| Planning under a budget 2,000 machines, two actions with known effects (a call and a visit), budget 8,000 for 400 machines | Expected benefit 22332 ± 955 planned, 6800 ± 1354 random; 93% of the best possible plan | The plan sends visits where they work and calls elsewhere |
| Checks on proposed actions 300 proposals: a helpful call, a harmful overtime action and an unmeasured part replacement | Harmful allowed: 0 of 495; unmeasured held: 505 of 505; helpful allowed: 265 of 500, blocked 0 | Harm is blocked and the unknown goes to a person; helpful actions pass only in runs where their measured effect was clear, and are otherwise held for a person |
| Features from the menu The 240-asset plant without the hand-written plugin | Skipped-maintenance count kept 5/5; AUC 0.778 ± 0.034 tuned vs 0.735 ± 0.059 untuned; tuned promoted 3/5 | The search finds the feature a person wrote by hand |
| Several possible outcomes 600 machines; repair, replacement or nothing within 60 days | Macro AUC 0.803 ± 0.011 vs 0.686 ± 0.022 for each machine's past mix | Recent repairs predict a replacement; the past mix does not |
| Why a rate changed Plant B's rate doubles while plant C's share doubles | Only B named as a driver: 5/5; C's mix effect 0.030 ± 0.001, rate effect -0.002 ± 0.002 | Rate and mix effects separate exactly |
| Pattern search 400 orders with two planted patterns (online orders rarely include a drink; pizza orders are worth more), and 400 with none | Planted found 2.0 ± 0.0 of 2; other findings 0.4 ± 0.5 per run; findings in random data 0.0 ± 0.0 | 126 comparisons per dataset under false-discovery control at 10% |
| Mapping validation 2,100 correct mappings across 15 domains | Correct mappings altered: 0 | Validation never alters a correct mapping |
Effect of a domain plugin
Real data: Online Retail II
Repeat purchase on the UCI Online Retail II dataset (CC BY 4.0): 1,067,371 invoice lines from a UK online retailer, 2009 to 2011. Cancellations and lines without a customer are removed, and lines are combined into 36,975 orders from 5,881 customers. The task predicts whether a customer orders again within 60 days, with point-in-time features and 20% of customers held out. The baseline is the recency rule. Recency baseline AUC: 0.761.
| Learner | AUC | 90% interval | Gain over baseline (90%) | Time |
|---|---|---|---|---|
| Linear (logistic) | 0.807 | 0.790–0.824 | +0.036 to +0.053 | 7 s |
| Linear with the survival feature | 0.807 | 0.790–0.824 | +0.037 to +0.054 | 7 s |
| Boosted trees | 0.812 | 0.795–0.827 | +0.043 to +0.059 | 38 s |
| Auto (chose boosted) | 0.812 | 0.795–0.827 | +0.043 to +0.059 | 93 s |
| scikit-learn logistic regression (reference) | 0.807 |
One dataset and one task. The scikit-learn row uses the same features and split and confirms the linear learner's
implementation. Boosted trees score slightly higher; the automatic choice, made by 3-fold cross-validation on the training customers,
picked BoostedClassifier.
Reproduce with python benchmarks/public/online_retail.py --data online_retail_II.xlsx.
Reproduction
python benchmarks/run.py --seeds 5 # writes benchmarks/results/core.json
python benchmarks/collect_lm.py --lm lm # summarises scored Vertiq LM runs into results/lm.json
python docs/build.py # rebuilds this site and the README