Training and evaluation methodology
All tasks follow three rules: features use only information available at prediction time, a fixed share of entities is excluded from training, and a candidate model replaces the baseline only when it performs better on those entities.
Point-in-time features
Entity-level holdout
Approximately 20% of entities, selected by a hash of their identifier, are excluded from training. No entity appears in both training and evaluation data.
Rule-based baseline
Event tasks are compared with a recency baseline and value tasks with the entity's previous value. Both scores are reported with every training run.
Champion–challenger comparison
A retrained model must perform at least as well as the production model on the same held-out entities before it replaces it, and its gain over the baseline must be clear at 90% confidence, from a bootstrap over the held-out entities.
Causal effect estimation
Statistically tested insights
Vertiq evaluates many candidate patterns at once, such as the day of a first purchase, first-order value and first-order products. Each candidate receives a p-value, and the set is filtered with the Benjamini–Hochberg procedure at a 10% false-discovery rate. Segments smaller than 20 entities are excluded. In the benchmark, the planted pattern was detected in 5/5 runs, with no false product patterns reported on random data.
Recursive self-improvement
Vertiq improves its models through recursive self-improvement (RSI) under evaluation control. In each round a teacher model generates additional training examples, and every example is validated against its own data before it is used. The student model is trained and evaluated on held-out records; it is promoted only if it outperforms the production model, and the promoted model becomes the starting point for the next round. Its incorrect outputs are retained as negative examples for the following round, and new outcome data is added at every cycle. No iteration reaches production without passing the evaluation gate.
Checking a mapping against the data
A mapping, whether written by a person, proposed by Vertiq LM or proposed from the values, is checked against every row before use. Summary and mostly empty rows are named and skipped. Identifiers that stand for "no value" ("-", "NA", a default number on many rows) become one-off entities instead of merging records. A role whose column is empty, a placeholder or the wrong kind moves to the clearly best-fitting column or is flagged; an amount column that is zero where another column records the same amount, or that amount times a quantity, gives way to the complete one. Judgements that need volume wait for 30 rows. Across 10,500 known-correct mappings, these checks changed 0.
Finding patterns without being told where to look
Every attribute value, group, weekday, part of day and item group becomes a feature. Each is compared with the record's value and with every other feature, and all p-values are filtered together with Benjamini–Hochberg at 10%, so the more comparisons are made, the stronger a pattern must be. A pattern must also be large enough to matter. On planted data the search found 2 of 2 patterns in every run and nothing in random data.
Improving itself under the gate
The tuning loop tries one change at a time (a learner, a setting, or a feature from a fixed menu of counts, recency and shares), scores it by five-fold cross-validation on training entities, and keeps it only if the mean improves and most folds agree. The final model must then beat the baseline and the untuned model on held-out entities. No generated code is run.
Effects of actions
With a control group: uplift for any recorded action, with cross-fitted checks. Without one: matched cohorts and doubly robust estimation, with balance and overlap reported. Live comparisons use sequential tests that can be checked after every observation without inflating false alarms.
Plans and checks on actions
Plans choose an action per entity within a budget from measured effects. Proposed actions, from a person or an automated agent, are allowed only when their measured effect is clear; unmeasured actions are held for a person.
Entity tables and leakage
Customer lists and asset registers join events by key. Their fixed labels describe and predict; totals and dates computed when the file was exported describe only, since using them as features would leak the future.
Testing on data it was not built for
Synthetic benchmarks plant a known answer. Mapping is also tested on six public exports from six sectors, held out from all training, with every acceptable answer listed per role; Vertiq LM gets 83% of roles right after validation. New behaviour found on real files becomes a general rule with a test on generated data, and the check that no correct mapping is changed runs after every change.