Vertiq LM
Vertiq LM maps an unfamiliar table to Vertiq's data contract. It identifies the entity, timestamp, event type, value and grouping columns, the attributes to retain and the date format. It is Qwen 3.5 4B (Apache 2.0) fine-tuned with LoRA on an Apple M1 Pro with 16 GB of memory, and it is evaluated on tables from domains excluded from training.
Results: original test set (114 tables)
| Model | Loads | Exact | Loads, validated | Exact, validated | Unseen, exact | Unseen, validated |
|---|---|---|---|---|---|---|
| Qwen 3.5 4B, no training | 57.9% | 3.5% | 97.4% | 9.6% | 4.2% | 6.2% |
| + LoRA fine-tune (round 1) | 66.7% | 20.2% | 95.6% | 31.6% | 12.5% | 20.8% |
| + ORPO on its own mistakes | 78.1% | 28.1% | 98.2% | 38.6% | 22.9% | 33.3% |
| Round 3 fine-tune | 93.9% | 35.1% | 99.1% | 40.4% | 29.2% | 33.3% |
| Round 3 + ORPO on its own mistakes | 85.1% | 52.6% | 99.1% | 64.0% | 43.8% | 58.3% |
Results: real exports (held out)
Six public exports from different sectors (animal care, city services, inspections, permits, retail, road safety), never used for training. Each role lists every acceptable answer; every proposal is validated before scoring, as in use.
| Method | Roles correct | Tables exactly right |
|---|---|---|
| Rule-based proposer (no model) | 57% | 0/6 |
| Qwen 3.5 4B, no training | 63% | 1/6 |
| Vertiq LM (round 3 + preference training) | 83% | 2/6 |
Results: plain-language rewrites (50 held-out descriptions)
A rewrite counts only if every number is the template's, each number stays next to what it described, and the rewrite is no longer than the template; accepted rewrites that are near-copies add nothing. Training data: 208 rewrites by a local 9B teacher, kept from 1376 attempts.
| Model | Accepted | Real rewrites | Near-copies |
|---|---|---|---|
| Qwen 3.5 4B, no training | 39 of 50 | 17 | 22 |
| Qwen 3.5 4B + LoRA on teacher rewrites | 30 of 50 | 27 | 3 |
Results: production-like test set (114 tables)
Includes audit columns and locale-specific number formats (R$ 1.267,81, € 4 350,50, ₹1,267). The earlier model is evaluated with its original prompt format.
| Model | Loads | Exact | Loads, validated | Exact, validated | Unseen, exact | Unseen, validated |
|---|---|---|---|---|---|---|
| + ORPO on its own mistakes (round 1 model) | 80.7% | 29.8% | 98.2% | 34.2% | 14.6% | 20.8% |
| Round 3 fine-tune | 91.2% | 41.2% | 100.0% | 44.7% | 31.2% | 33.3% |
| Round 3 + ORPO on its own mistakes | 93.0% | 46.5% | 100.0% | 53.5% | 37.5% | 41.7% |
Findings
Adding column types, inferred from the values, to the prompt raised exact matches on unseen domains from 14.6% to 31.2%. Inferring date formats and value types from the rows improved every model.
Preference round
ORPO on the round 3 model's own mistakes raised exact matches after validation from 44.7% to 53.5%, and on unseen domains from 33.3% to 41.7%. To fit 16 GB, the 8% of preference examples longer than 460 tokens were left out. DPO was not used because it keeps a second copy of the model. Constrained decoding was evaluated and not adopted: it would correct 2 of 88 remaining errors.
Several Macs
lm/mesh.py sends LoRA sweeps (one job per setting) and evaluations split into
shards to a job-queue head node; workers on other Apple Silicon Macs run them while those Macs are idle and on wall power. Loss,
validation loss and peak memory stream to the head's dashboard. A job saves its adapter every 25 steps and resumes on another Mac
if its own is needed. Only the vertiq package, the job scripts and the named data go to a worker.
Reproduction
lm/make_data.py, lm/train.sh, lm/evaluate.py,
lm/collect_failures.py and lm/pref.sh; lm/round3.sh runs the production-like round end to end.