Tabular foundation models

The “in-context learning for tables” lineage: a pretrained Transformer reads a labeled table as its context and predicts labels of new rows in a single forward pass — no training, no tuning, no feature engineering per task. The shift LLMs demonstrated for text, applied to the data format enterprise ML actually runs on (churn, default, demand, price — the gradient-boosted-tree kingdom for two decades).

The mechanism (from the Kumo Tabular post, captured 2026-09-30)

Three attentions (TabICL/TabPFN lineage): column attention (what a value means in its column’s distribution — is 42 typical?), row attention (how features interact; rotary column positions; 4 learnable [CLS] readouts compress each row), in-context attention (query rows attend to context only — context KV computed once and reused for follow-ups; Test-GQA shrinks the read cache). Notable detail: length-aware attention temperature — softmax dissolves over tens of thousands of keys, so every query scales by a per-head temperature growing with log(keys), keeping attention sharp as tables grow. Regression emits 999 quantiles → point prediction + uncertainty.

The boldest choice: pretraining entirely on artificial tables — a Structural Causal Model sampler draws causal graphs, random mechanisms (linear/NN/tree/GP), hidden confounders, missingness patterns, heavy tails, coarsened duplicates; a tree-ensemble filter discards tables with no learnable signal. Procedural sampler = infinite tables; the model learns mess-handling without cleanup. (Same prior-data philosophy as TabPFN’s synthetic prior.)

The field

• Kumo Tabular (NVIDIA, post captured; weights on HF, code NVIDIA/structured-data-models, OpenMDW-1.1) — 28M/65M/215M sizes, ranks first on TabArena (ELO 1950, 17× faster than LimiX-2 under uniform RTX 6000 Pro), BeyondArena, TALENT, ScoringBench. Limits: numerical+categorical only (text/timestamps via preprocessing recipes), 10 classes per pass (ECOC extends), degrades on out-of-range tables or query/context distribution shift.
• TabPFN / TabPFN v2 (PriorLabs) — the lineage’s origin: prior-fitted networks, ICL on synthetic priors. TabICL (soda-inria) — scaled-context successor. LimiX-2 — the efficiency rival Kumo benchmarks against.
• GBTs don’t die — tuned XGBoost/LightGBM remain the accuracy floor every one of these must beat without tuning.

Why it matters here

• Same “pretrained beats per-task pipeline” argument as Decision models — but for typed tables instead of text: schema-in, calibrated probabilities out, zero per-task lifecycle. The two clusters rhyme: decision models = LLM→structured decisions; tabular FMs = GBT→in-context predictions.
• Practical homelab/work angle: oncall/inventory-style tabular datasets (counts, statuses, categorical labels) are exactly the small-table regime where ICL tabular models shine — a zero-training classifier for ad-hoc “does this pattern predict that outcome” questions.

Open

• (open: hands-on — Kumo Tabular-Small or TabICL on one oncall-style table; the 28M size runs on CPU-class hardware)
• (open: does the SCM-synthetic-prior trick transfer to other structured domains — logs, metrics? — or is it tables-specific?)

Related: Decision models · Model blends