I didn’t attend, but still marked the event in my calendar to check on current trends in tabular foundation research. Going through the titles and some abstracts was very informative, and I want to share my observations about the state of tabular foundation research.

If you are new to tabular foundation models, start with this series:

Here are the trends and observations, based on this list of papers:

• Tabular foundation models are expanding beyond just regression and classification, as I expected. There are papers on survival analysis, Bayesian inference, causal identification, and more.

• Time series was the biggest ML task/modality (besides tabular) with around 50 out of 151 papers. I attribute it to the workshop framing (“tabular and time-series”). By the way, I just published a book chapter on time series forecasting with tabular foundation models.

• Tabular foundation models are moving away from being just models toward becoming more of a system, with research on external infrastructure such as context management, input-space adapters, distillation, and integration into agents.

• Benchmarking and evaluation were a huge topic. Also, some critique about the current, narrow way of benchmarking. I personally think this is improving thanks to new benchmarks like ScoringBench and BeyondArena.

• Some researchers examined prior designs such as causal DAGs and graph priors.

• An emerging topic is mechanistic interpretability for foundation models.

• If you are curious about the application of tabular foundation models, you’ll find some applied papers in the list as well, in domains like healthcare.

• Some papers explored multi-modal foundation models, like tabular+text and tabular+image.

Some topics were surprisingly underrepresented:

• The focus was not as much on scaling foundation models, at least compared to industry labs, where scaling tabular foundation models to larger data and speeding up inference is top priority.

• There was a bit on relational foundation modeling (extending foundation models to a relational multi-table setup), but I expected more here.

Papers that caught my interest: Beyond Accuracy: Toward Trustworthy Tabular Foundation Models in Industrial Applications and Exploring Differences Between Tabular Enterprise Data and Public Benchmarks.

For me, this time is now.

In 2022, I made the scary decision to become self-employed as a writer for technical books. With the Interpretable ML book already under my belt, some financial runway, and a support network, I gave it a try. I didn’t want to look back at my life and regret never trying the author life. There was a real risk, since most authors can’t live from writing books. Fortunately, I managed to make a living from just books, and I am super grateful to everyone buying my books, making it possible for me to become a full-time author.

At some point, though, I felt my world shrinking due to the self-inflicted constraint of saying no to everything that is not about writing. So I added more projects to the mix: I gave workshops on ML topics like interpretability and uncertainty quantification, and I consulted clients. At one point, I even (successfully!) invested 3 months into a machine learning competition to prove to myself that I’m still in the data science game and not just an impostor. Adding projects next to writing temporarily helped.

But now it’s time for a bigger change. I miss being part of a team, working together on a shared goal and projects. I miss ongoing exchanges with colleagues, the deadlines (yes, really), and this certain randomness (in the positive sense) you only get when being part of something bigger.

What’s next

I don’t have a concrete job lined up right now, but I started applying and having first conversations. I have no idea where I will end up yet, and that’s quite exciting.

If you would like to work with me or know of a job opening in the ML/data science space, I’d be very happy to hear about it! Just reply to this email, or send an email to chris at christophmolnar.com.

What it means for Mindful Modeler: I will continue posting here, but maybe on a more irregular schedule. Long-term, my writing for Mindful Modeler depends on what job I will have, whether it’s part-time or full-time. For some jobs, it might even make sense to post as part of the job itself.

I’ll continue writing the Tabular Foundation Models book during the job search. It’s probably going to be a bit shorter, focusing on the essentials. I already published a few chapters; check the tabular foundation book out here.

I’ll forever be grateful for having been able to write full-time. Thanks again to everyone who bought my books and followed along on my journey, here and elsewhere!

The AI influencers found out, and social media is now awash in TabFM posts.

Let’s talk about what TabFM is and isn’t, without the hype.

TabFM is a tabular foundation model with in-context learning

TabFM is a tabular foundation model like TabPFN, TabICL, and TabDPT.

More technically, TabFM is a transformer-based neural network pre-trained on hundreds of millions of synthetic datasets. TabFM makes predictions via in-context learning: there is no classic training step, but the training data is provided during inference time and serves as the context to predict the test data. It can do both regression and classification (up to 10 classes).

If you want to learn more about tabular foundation models in general, read my TFM series:

TabFM follows in the footsteps of especially TabPFN and TabICL, which the authors note in their research announcement post. It re-uses architecture elements of both:

• For example, TabFM uses alternating row and column attention, as e.g. TabPFN 2 did.

• TabFM uses row compression and performs ICL over the compressed rows, as TabICL does.

Also, TabFM is pre-trained on hundreds of millions of synthetic datasets generated with structural causal models, just like TabICL and TabPFN. However, not much is known about pretraining and prior, because the published code is just the inference code, and there is no white paper or paper published, just an inference code repo, the research announcement post, and the Hugging Face model (weight) release.

Being a tabular foundation model, TabFM inherits the “standard” pros and cons of modern foundation models: No tuning needed; highly performant; slow inference; need to provide training data at inference time; and so on.

So, TabFM is not the first tabular foundation model, nor the last. What is all the hype about?

One of the reasons for the hype: TabFM climbs to the top of TabArena, a benchmark for (primarily) tabular foundation models. If you want to learn more about TabArena, I've got you covered:

TabFM’s entry is not yet reflected in the live leaderboard at the time of writing, but results are waiting to be included in a pull request. These results are certainly impressive.

So, should we all be using TabFM now? I won’t be, for now, for two reasons.

Slow inference and non-commercial license

TabFM is larger than the other tabular foundation models. For example, it defaults to 32 estimators, when most other TFMs have a default of 8. Also, many aspects of the architecture are scaled up: TabICL contains 4 CLS tokens; TabFM contains 8. TabICL has 12 ICL transformer blocks; TabFM has 24. TabICL has an embedding size of 128; TabFM has 256. In many ways, TabFM is larger than the current 2nd-generation (TabICL v2.0; TabPFN-3.0).

This scaling up and the alternating row and column attention come at a price: Inference is slower than for the other TFMs. The maximum number of features is 500.

The bigger issue no one talks about is the license. If you go to their GitHub repo, you see an Apache-2.0 license. Great news, right? But that repository only contains the inference code. The weights are downloaded from Hugging Face and come with the tabfm-non-commercial-v1.0 license, which prohibits commercial use. In my understanding, if you want to use it commercially, you would have to contact Google and ask for permission.

For new TabFM users, I don’t think it’s obvious: At the time of writing, the non-commercial license is not mentioned in the README, and the weights are downloaded silently from Hugging Face. See also this GitHub issue. I hope Google addresses this; otherwise, users might unknowingly violate the non-commercial license.

All in all, TabFM's performance seems impressive, and I am looking forward to seeing how it holds up in other benchmarks. The model seems to be geared towards performance, at the cost of inference speed. The non-commercial license makes it less attractive for me.

So, what is the best model (class) to use?

The short answer: Second-generation tabular foundation models like TabICL v2 and TabPFN-3.0.

These pre-trained models are capable of outputting the full predictive distribution. Meaning you get the quantiles “for free”. Ignoring that TFM inference is kind of expensive, there is at least no additional cost.

Evidence from ScoringBench

The evidence for TFMs being good at predicting beyond the mean comes from the benchmark ScoringBench (website|code|paper):

• This benchmark compares models based on proper scoring rules and other metrics that evaluate the entire predictive distribution.

• An example of a simpler metric of the benchmark is the coverage of the 5%–95% prediction interval (which should be 90%).

• An example of a proper scoring rule in the benchmark is the Continuous Ranked Probability Score (CRPS).

The following figure shows the ranks of various ML models/algorithms:

But what exactly are the fine-tuned models that lead ScoringBench? Since these PFN-based tabular foundation models like TabICL and TabPFN are “just” neural networks, they can be fine-tuned. And if you want them to become better at predicting such scores, you can specifically fine-tune them for scoring-rule objectives. However, even there, it matters which metrics you fine-tune. From the paper: “TabICLv2 was fine-tuned with the CRPS objective, so it improves on CRPS but not on the log score, consistent with the expectation that fine-tuning shifts a model’s inductive bias toward the optimized scoring rule.”

Two caveats

The differences between tabular foundation models and the other ML algorithms seem large in the chart. But they are only large in terms of median ranks. The actual effect sizes are only negligible to small. However, paired with their often stronger performance, tabular foundation models are a great deal, if you can swallow the higher inference cost.

Even fine-tuning doesn’t guarantee coverage; it at best approximates it. Although it seems that TabPFN and TabICL are already pretty good at non-mean predictions. If you need (marginal) coverage guarantees, you would have to go with something like conformal prediction. If you are interested in that, I have a book for you: Introduction to Conformal Prediction With Python.

Anyways, if you need quantiles or anything beyond mean prediction, give tabular foundation models a try.

TabBench for classification tasks

We take a look at the TabBench V2 benchmark, which evaluates ML algorithms on 189 classification tasks. These classification tasks range from one hundred up to 150k rows, from binary to up to 100 classes, and from 2 to 1777 features. All datasets are IID classification datasets from OpenML. The company behind TabBench V2 is NeuralkAI, based in Paris, France.

TabBench compares three “clusters” of ML algorithms: tabular foundation models (TFMs), gradient boosted decision trees (GBDTs) like CatBoost, and tuned neural networks (NNs) like TabM.

Overall win rate of state-of-the-art (=2nd generation) Tabular Foundation Models: ~83% (caveat: based on accuracy; but TFMs are highly performant under the other metrics like F1 as well). The list of 2nd-generation TFMs includes TabICLv2, TabPFN-3.0, and Seldon.

Further down on the benchmarks website, there is a quite interesting figure, which drills down performance based on dataset characteristics.

As far as I understand, the axes are the first two PCA components of the datasets × model-metric rank matrix. The x-axis goes from small tasks with mostly numerical features to categorical-feature-heavy, large tasks; the y-axis goes from binary, balanced tasks to multi-class, imbalanced tasks.

Three distinct clusters emerge, which give us a rough idea of when to expect which ML algorithm to work well.

When does which ML algorithm perform well?

Based on that same data, the creators of TabBench also provide a decision tree (I assume they trained it on the results), which gives us a good rule of thumb when to expect which ML algorithm to perform the best classifications:

• Small to midsize (<44k rows) classification dataset with few categorical features (<15%) → Use Tabular Foundation Models (2nd-gen)

• Smallish datasets with many categorical features → Use boosted trees

• Larger binary datasets (>44k rows) → Use boosted trees

• Large multi-class or unbalanced dataset → Use tuned neural networks

These are, of course, just tendencies and not physical laws. Your task may be balanced, binary classification, and your best model might be a neural network, or even a logistic regression model. By the way, logistic regression was not part of the benchmark, which would have been interesting to see as a baseline. Current baseline is ensembled xgboost. Another caveat: The benchmark only goes up to 150k rows, so beyond that, we can only guess. My guess, though, is that beyond 150k rows, tabular foundation models will not (yet?) take home too many prizes.

Where does that leave tabular foundation models? This snapshot supports the view that tabular foundation models are state-of-the-art models that stand side-by-side with boosted trees and tuned neural networks. However, the development of TFMs still has a lot of momentum. Go back to December 2025, and the recommendation would have been something like: use tabular foundation models only for very small datasets. Since then, they have been eating into the boosted tree territory improvement by improvement. The question is: how far will the TFM territory expand? I’m very curious to see how the TFM field will further develop, and, of course, will share with you what I learn along the way.

In the meantime, I encourage you to take a look at the TabBench V2 results and play around with the interactive figures; the site contains many more insights than I covered here.

If you want to learn more about Tabular Foundation Models, have a look at the open book I am working on: tabularfoundationmodels.com.

So, what is this mysterious Thinking mode?

Thinking mode is not open source

Not much is publicly known about this mysterious Thinking mode. It’s bundled into “TabPFN3-Plus,” the closed-source/commercial part of TabPFN. The technical report has only one paragraph on the Thinking mode and keeps things high-level. In general, Prior Labs has chosen a dual path: an open core with their model architecture being open source plus closed source parts, like the prior and the Thinking mode. This is a similar playbook to what open-core infrastructure companies like Red Hat and GitLab are doing. Give away the core for free, monetize some of the features, especially for enterprise.

What we know about Thinking mode

We have two sources available to go full detective. Let’s start with the TabPFN-3 Technical Report, which gives us the following hints:

• Thinking is “test-time compute scaling” of TabPFN.

• The authors also refer to the mode as applying “additional inference-time computation.”

• It outperforms AutGluon 1.5 extreme in less than a tenth of the runtime.

• No use of LLMS, real data, internet, or any other model besides TabPFN.

• The performance gains seem to be mostly for large datasets.

A bit more information is provided in the TabPFN documentation:

• The Thinking mode flag is set in .fit(), not in .predict(). The docs also state that the Thinking mode “fuels recurring predictions”.

• There is a thinking_effort parameter, which can be set to 'medium' or 'high'. This parameter steers the “effort & compute” of the Thinking mode.

• The thinking_timeout_s parameter sets a time budget.

• A “thinking_metric” parameter controls which target metric to optimize during Thinking mode. Currently supported metrics are: Accuracy, LogLoss, ROC AUC, RMSE, and MAE.

• There is a separate monthly quota for Thinking mode of 20 fits per month.

Alright, this is all I found. Everything that follows is speculation based solely on the limited publicly available information.

What Thinking mode is probably not

Even though not much is known, I would exclude some possibilities for Thinking mode based on the available information.

• I don’t think it’s an ensemble with other models, since the report states that no other models were used. However, it might be an ensemble of TabPFN models.

• It’s probably not some novel pretraining procedure, as TabPFN-3 with and without thinking seem to share the same pretraining.

• No LLM chain-of-thought involved, as the report states no LLMs involved.

• I don’t believe Thinking mode involves gradient-based fine-tuning of TabPFN, since the metrics the user can pick are not differentiable.

Let’s speculate

The TabArena results for plain tabular foundation models are quite strong. And yet, they “only” do in-context learning, which does not even change the model weights. Besides that, no particular optimization happens in vanilla TFMs. For example, TabPFN is already an ensemble, but the ensembling is agnostic of the underlying task: The variety between ensemble members comes from changing the feature order and the pre-processing pipeline. What I want to say: there is an opportunity to optimize beyond in-context learning.

In my opinion, there are strong hints that Thinking mode might involve some validation-driven search: there is some computational expense (up to 1/10th of the AutoGluon 1.5 extreme compute time); the budget parameters would be coherent with some type of search with a budget (thinking_effort and thinking_timeout_s); also, the user can pick a metric, which might be the objective of some optimization; It would also make sense given the team’s AutoML background.

If we assume that Thinking mode is search with a budget, the big question would be: What is the search space? Does it include feature engineering? Is it about smart context selection? Is it maybe a more task-specific way of constructing a TabPFN ensemble? The gain in performance (in Elo) is quite large, so I’m assuming that it’s optimizing multiple things, not just a bit of feature engineering or so.

Another possibility is that Thinking mode does some type of context distillation or optimization, which might explain the strong gains on large datasets.

I am also wondering whether the Thinking procedure is model-agnostic. And if it is, would it give the same gains to xgboost and co, or is there something about tabular foundation models that makes the Thinking mode more effective?

These are my thoughts so far. I know, I am leaving you with more questions than answers 😅. What are your guesses?

Let’s find out.

How TabICL handles missing values

For TabICLv2, we can find that information in the README, section “Preprocessing”:

• Create a separate category for missing values in categorical features

• Perform mean imputation for missing numerical values (encoded as NaN)

This looks like a pre-processing layer that you could attach to any model. Creating a missing value category is pretty standard. Mean imputation for numerical values is something I see often, too, unfortunately (mean imputation loses a lot of information).

A look into the TabICLv2 paper confirms that missing values handling is indeed just a pre-processing layer:

Adding missing indicators (Le Morvan & Varoquaux, 2025) or introducing missingness during pretraining may improve the handling of missing values, which are currently imputed by the mean, but remain unexplored.

My recommendation for TabICL, in this current version: Deal with missing data imputation yourself. Or at least confirm that adding a missing value category for categorical features and mean imputation for numerical features is what you want.

How TabPFN handles missing values

Let’s have a look at TabPFNs README to see how it handles missing values:

Q: Can TabPFN handle missing values?

Yes!

Ok … cool, I guess. Not very informative though.

Let’s go deeper. The technical report for TabPFN-3.0 says:

Native missing-value handling. For each cell that is NaN, TabPFN-3 computes a binary indicator and concatenates it with the cell value before embedding. The model therefore receives an explicit signal about missing data and can condition its predictions accordingly, rather than relying on upstream imputation.

This sounds promising. Let’s go even deeper.

In the TabPFN code, we can see that missing values get encoded and passed into the model: The NanHandlingEncoderStep class encodes NaNs as -2.0, Inf as 2.0, and -Inf as 4.0. The features themselves are then mean imputed, but the additional “missingness” channel is concatenated with the original features before the linear embedding. It gets a bit more complicated because features are grouped, but that’s not important right now. We only need to know that the TabPFN-3.0 model has the information on missingness for each feature available during inference time.

Having these missingness indicators in the architecture is only half of the story. The other part is pre-training with missing values. Unfortunately, TabPFN’s pre-training code is not open source, but I guess that TabPFN was pre-trained on datasets with missing values.

Based on the way that TabPFN’s architecture encodes missingness, it is well equipped to deal with missing values. According to this research paper, TabPFN can, in theory, even handle the most difficult type of missingness: missing not at random (MNAR). However, I haven’t tried it out myself so far and have not seen any benchmarks yet.

One answer is tabular foundation models like TabPFN and TabICL. They are prior-data fitted networks, an approach to tabular foundation models. I covered the PFN family extensively in this blog (start here). In short, TabPFN and TabICL are transformer-based networks, pre-trained on millions of synthetic datasets, allowing for in-context learning.

But what about other approaches to learning from some tables and transferring them to another table?

Besides PFNs, there are hypernetworks, cross-table transfers, and LLM-based approaches.

This post zooms out a bit, looking at these other tabular foundation model families.

Hypernetworks

Hypernetworks are pre-trained to “predict” or emit weights of a smaller model, typically a multi-layer perceptron (MLP), which is then used to make the actual prediction. The “prediction” step of the hypernetwork, therefore, replaces the training step of the MLP. And the actual prediction for your current data is then just regular inference with an MLP. Hypernetworks are often based on transformers (like MotherNet), but don’t have to be (for example, iLTM)

Cross-table transfer plus fine-tuning

This family follows the more classic foundation model recipe with self-supervised pre-training across many tables, then fine-tuning on your current data. The pre-training task is usually self-supervised, for example, a masked reconstruction task, where some cells are hidden, and the model predicts them. Approaches in this family have in common that they learn to encode diverse tables into a schema-agnostic representation, and they are often transformer-based.

During pre-training, each table might get its own input/output adapters (featurizer and head) that are discarded for inference. Instead, to apply the model to new data, you attach a fresh set of adapters and fine-tune.

The family holds a diverse set of models, differing mostly in how to encode tables into a schema-agnostic representation. For example, some approaches also encode semantic information such as column names (e.g., CARTE), while others don’t (e.g., XTab).

LLM-based approaches

The basic idea: take a table, turn it into text, then fine-tune an LLM in a supervised fashion. This reframes table prediction as a language problem to force it into the LLM scheme. I find that the weirdest family, as LLMs have the wrong inductive biases for tabular data, especially when the tables have lots of numerical values. On top of that, inference cost is relatively hefty. The LLM-based family may be most interesting for small tables with lots of semantic features, like free text features, categories, and telling feature names. This TFM family is the only one that allows zero-shot predictions, but can also do few-shot learning (aka In-Context Learning). An example is TabLLM.

Why I focus on PFNs in my book

Part of what brought me to write this post is thinking about the scope of my latest book project, Tabular Foundation Models.

The book is focused on just one of the families, the PFNs. So why not cover these other families in the book? Here is my reasoning for why I’m betting specifically on PFN-based models like TabPFN and TabICL:

• PFNs are currently state-of-the-art, even beating boosted tree models.

• Great open source availability and ecosystem.

• There is a lot of development happening in the PFN space.

• Many startups and companies are betting on PFN-based models.

• PFNs allow for in-context learning, no further training required.

• No fine-tuning required. But you can fine-tune if you want to.

• It’s easy to instill the right inductive biases, as you can include synthetic data in the pre-training.

So, to me, PFN-based approaches like TabPFN and TabICL hold the greatest promise for foundation modeling. That’s why I’m betting that they’ll end up as a new state-of-the-art paradigm in tabular modeling.

In terms of book writing, my last year has been horrible. I quit writing Building Blocks of ML after months of work, and I put Machine Learning 4 Remote Sensing on ice. My 2025 experience has made me wary of starting new book projects. So I focused on Mindful Modeler, especially the series on tabular foundation models.

I enjoyed writing the TFM series a lot, and it resonated with many of you. This truly built up my motivation and confidence, and I started writing a book again:

Tabular Foundation Models — A Hands-On Guide to TabPFN, TabICL, and the Tabular Revolution

There is such a huge gap between frontier labs pushing tabular foundation models and everyday modeling practice. Many data scientists I’ve talked to have barely heard about tabular foundation models, struggle to learn about them, or have had strong concerns with explainability and scaling. There is just so little coverage of foundation models: barely any tutorials, videos, posts, or books.

So, here is my plan.

My plans for the Tabular Foundation Book

I want the book to be the best introduction out there on tabular foundation models, covering the three parts: intuition, theory, and practice.

The book will be open, meaning a web version for anyone to read in the world, free forever. Just like I did with my book Interpretable ML and Supervised ML for Science.

Model-wise, I will likely focus on TabICL and TabPFN. While there are many more models and alternative approaches to the foundation model paradigm, the book’s priority will be practicality and performance, and not academic coverage. So more of a book for cooks, less for botanists.

It will be a challenge to keep up with the fast pace of the tabular foundation field. That’s why I’ll put extra effort into making the book as “timeless” as possible. I want the book be worth your time investment.

The good news is that I am fairly confident that code examples should age well, since TabICL and TabPFN adopted the scikit-learn interface. I’m planning to put many practical Python code examples into the book. From simple classification and regression examples, to time series forecasting, quantile regression, speeding up inference, imbalanced classification, … you name it.

I plan to move fast with the book, get an early version v1.0 ready by autumn (fingers crossed), and then keep updating it from there on.

That’s my plan. Only one piece of information is missing …

Which cover mascot?

All my books have had a mascot on the cover: A mole, an octopus, a beaver, meerkats, and a raven. For the current book, I am looking for a cover mascot that captures the idea of foundation models. Some candidates:

• A chameleon would represent “one-shotting” patterns.

• A mouse that can live in any environment.

• Honeybees working on a grid of cells, building a foundation.

If you have any ideas, feel free to leave a comment. The book is still in progress, which means I’d love to hear your feedback and requests for chapters.

But the tabular benchmark situation is changing; a change that goes hand-in-hand with the rise of tabular foundation models.

Changes are two-fold: Benchmarks are becoming more “live” and focused on “capabilities” at least from the lens of tabular foundation models.

Live benchmarks, like TabArena, are typically very rigorous with a strict protocol for standardized pre-processing and evaluation. But what makes them “live” is that they come with a website and active maintenance, reflecting the current state-of-the-art. This is in contrast to static benchmarks, which may be a table in a PDF paper, without updates. If you want to learn more about TabArena, I have a full blog post:

Live benchmarks are quite common in LLM development, where we have SWE-bench for assessing coding, LongBench for testing LLMs with long contexts, and Humanity’s Last Exam with a list of expert-level questions. Each benchmark addresses different “capabilities” of large language models.

Benchmarking capabilities in tabular ML

Tabular is, by nature, a narrow modality compared to language, which is more general-purpose (translation, coding, question-answering, …). However, even the tabular modality contains many tasks: classification, regression, quantile regression, missing data imputation, time series forecasting, and many, many more. If someone designs a new ML algorithm, they can benchmark it against any of these tasks (if the algorithm is flexible enough).

The novel appeal with benchmarking tabular foundation models is their even greater flexibility, and that we are not testing an algorithm, but a fixed model.1 Whether you use TabICL for regression, quantile regression, or time series forecasting, it’s always the same pre-trained model, and due to in-context learning, the weights don’t change. This parallels LLMs, where we have pre-trained models with in-context learning. This invites us to reframe tasks as “capabilities.”

I’m excited about seeing a proliferation of benchmarks to test “capabilities” beyond just classification and regression. For example, ScoringBench evaluates ML algorithms and tabular foundation models based on their capability to predict the full predictive distribution.

While benchmarks have always been a catalyst in machine learning, it feels like the benchmark landscape for tabular ML is changing, due to tabular foundation models. For example, just recently, the MulTaBench paper was put on arxiv. The benchmark contains 40 multimodal datasets, 20 of which are tabular plus text, and the other 20 are tabular plus image. Exactly the catalyst we need to move forward on multi-modal tabular foundation models.

A piece of evidence pointing toward the new benchmark situation is the TabPFN-3.0 model report. Out of the 20 main pages, 9 are “Experimental Results”, mostly benchmarks. For example, they test classification, regression, and quantile regression capabilities on TabArena, and prediction with text columns on TabStar data, and relational data with RelBenchV1. Not all are “live” benchmarks, but they test different capabilities.

I remain excited about tabular foundation models. The tabular foundation models field already has a strong momentum, and having diverse benchmarks may serve as catalysts. However, there is the risk of overly focusing on benchmarks. Think overfitting and benchmarks-as-marketing, as we are seeing with LLMs. That’s why it’s important that we have many diverse benchmarks from various parties.

This post is a collection of tips and tricks to make tabular foundation models much faster. BUT! There is always a price to pay. And you must decide on that bargain. Some improvements are cheaper, some are more expensive.

If you haven’t heard about tabular foundation models, check out my series on TFMs:

Let’s start with a “cheap” option for improving the prediction time of tabular foundation models.

Pick a faster foundation model (currently TabICL)

There has been a flurry of new models in the TFM space. Not all have the same inference time. It may be worth comparing the speed of multiple TFMs.

Which one is the fastest TFM? This is subject to rapid changes, but here is a quick first idea, based on TabArena. If you don’t know TabArena, check out my post.

The following leaderboard excerpt shows that TabICLv2 is, on average, the fastest TFM, but not the most performant. The best-performing model is TabPFN-2.6, with a slightly better Elo than TabICLv2.

However, results from TabArena are averages (or medians) and therefore just rough pointers. You won’t know which TFM performs best on your task until you try.

I compared the prediction time of TabPFN and TabICL on the UCI breast cancer dataset: Switching from TabPFN to TabICL reduces inference time by ~14% (TabICL: 0.42 seconds; TabPFN: 0.49 seconds). TabICL (acc: 0.98, logloss: 0.067) even slightly outperforms TabPFN (acc: 0.97, logloss: 0.087) on this data.

Use a GPU

Tabular foundation models are neural-network-based and are much faster on a GPU than on a CPU. If you are GPU-poor like me, this sucks, as it requires getting access to a GPU. With boosted trees and other classic machine learning algorithms, I was able to run most projects on my good old MacBook M1. With tabular foundation models, a GPU is much, much more performant. Switching from CPU to GPU is one of the best levers to improve prediction time.

To give you an idea: Classifying the breast cancer dataset on a CPU with TabICL (on Google Colab) took ~15 seconds. Switching to GPU reduced that time to ~0.5 seconds, which is a 30x improvement.

Cache training representations

For each prediction call, tabular foundation models push the entire training data through the network. Let’s say you have two test sets for which you make predictions in two separate .predict() calls. For each of these calls, most computations are identical. That’s because the most compute-intensive parts are the transformer modules in the neural network. Here, training data can attend to training data, and test data can attend to training data. That first part, training-attends-training, is the same for each .predict() call, regardless of what the test data looks like.

A clear case for caching, right?

Super easy to do, here with TabICL:

clf = TabICLClassifier(kv_cache=True)
clf.fit(X_train, y_train)  
clf.predict(X_test)

With caching, the .fit() call stores the training data computations in a key-value store. The .predict() step can now run faster since it only needs to make computations for the test data and can look up all the attention values for the training data.

For the breast cancer data, this reduces the .predict() time from 0.5 seconds to 0.09 seconds for TabICL.

But there is a catch. Two, actually.

First, caching costs memory. A lot of memory. According to the TabPFN 2.5 paper, 6.1 KB of GPU memory and 48.8 KB of CPU memory per cell of the training data. So, in our case, for a small dataset of 455 rows and 31 columns, which means 14105 cells, we already have 86 MB of GPU memory and 688 MB (~0.67 GB) of CPU memory. And that’s for a small dataset. That makes caching more realistic for small data or for reduced contexts. For TabICL, I haven’t found the memory numbers, but I suspect they will be in the same ballpark.

The second catch: While .predict() is faster, .fit() is accordingly slower. So if you do a single train/test split and call .predict() exactly once with the same context data, the KV caching will be useless. Use caching only with repeated .predict() calls and when you actually do have the memory.

Context Optimization

Models like TabICL scale with O(n^2m + nm^2). Assuming the number of features m is much smaller than the total number of rows n, and context data dominating n, then halving the context data size may almost mean a 4x reduction in compute time. Any strategy that reduces context size substantially improves speed.

Reducing the context size usually means lower predictive performance. The following chart shows how simple subsampling with different sample sizes of the context data affects predictive performance, in this case for the UCI wine quality dataset.

There are also smarter ways to reduce context size, such as a k-nearest neighbor-based approach and data/distribution distillation approaches.

More details on context optimization in this post:

Since TabICL’s runtime of O(n^2m + nm^2) is also quadratic in terms of the features, reducing the number of features is also an option, especially if you start out with lots of features. To reduce the number of features, we have the entire toolbox available, from dimensionality reduction to feature selection.

Reduce the ensemble size

When you call tfm.predict(), you get the result from multiple TFM calls with slightly different parameters. This has to do with the dependence of TFMs on the column order.

The default ensemble size is 8 (coming down from a whopping 32 in the first version of TabICL). Hiding in there is therefore an up to 8x speedup. The speedup is usually at the cost of reduced performance, but not always. In the case of the breast cancer dataset, we get with TabICL:

• 8 estimators: 0.42s and a logloss of 0.0671

• 4 estimators: 0.22s and a logloss of 0.0659

• 2 estimators: 0.11s and a logloss of 0.0760

• 1 estimator: 0.06s and a logloss of 0.0717

In this case, accuracy even stays the same between 8 and 1 estimators. Here, reducing the ensemble to 1 estimator is justified and gives us a huge speed improvement. For your own data, I recommend verifying whether an ensemble reduction brings speed improvements.

Train a surrogate model

You may distill a TFM into a surrogate model, like a tree ensemble or a multi-layer perceptron. The distillation approach can make sense if you do a lot of inference with fixed context/training data or run it on constrained devices. But you lose all the benefits of using TFMs in the first place, especially the flexibility in context used, embeddings, and so on. Especially if TFMs move into a multi-modal and relational direction, which I strongly suspect, this line will be much less relevant.

Key speed improvements for the impatient

• Use a GPU.

• Switch to a faster TFM.

• If memory allows, use KV caching for repeated predictions with the same context.

• Let optimizations compound: For example, GPU inference plus KV caching can improve prediction time by well over 100x.

Also: Scaling is a high priority for most TFM labs. I expect further improvements in architecture and inference tricks, which will bring down the .predict() time.

In my post titled I’m betting on tabular foundation models, I argued that TFMs will work with all kinds of supervised ML tasks because you can pre-train for any task for which you can construct a data generator. This includes the task of time series forecasting.

Besides specific pre-training, there is another path: Reframe the forecasting task as a regression task and throw a tabular foundation model at it.

This post is about the reframing + TFM approach. This approach is called TabPFN-TS in this paper for TabPFN, but you can replace TabPFN with other TFMs, such as TabICL or TabDPT.

The idea is simple: Take a time series, automatically enhance it with temporal features based on the timestamp, then one-shot it with a TFM regressor.

The input can be as simple as a target plus a timestamp:

For this simple, univariate time series with equidistant time steps, we only need to provide the context data and the number of time steps to get a prediction, here with TabICL:

model = TabICLForecaster()
pred = model.predict_df(context_df, prediction_length=10)

Time-based features are automatically added. These include an index that simply counts through the timestamps, cyclical (sin/cos) calendar features for day of year, day of week, and a few more features.

Instead of using a vanilla tabular foundation model, you can also use a specific time series foundation model, such as TiRex, Toto, or Moirai-2.0. Based on the paper, the time series foundation models outperform tabular foundation models on univariate time series tasks. However, the leaderboard flips when introducing other features: TabPFN-TS outperforms the time series-specific models for covariate-informed forecasts.

So what?

Let’s disentangle this:

• Even if not pre-trained for forecasting, tabular foundation models like TabPFN work well as a backbone for time series forecasting.

• I found it surprising that tabular foundation models outperformed time series foundation models in multivariate settings.

• The wrapper that turns TabPFN into TabPFN-TS is not unique to tabular foundation models. At least to my understanding, it should also work with other “traditional” ML models.

• However, providing such a wrapper for tabular foundation models is very on-brand, as it further pushes TFM down the path of becoming a batteries-included, does-a-lot-of-things-for-you-by-default type of model.

We use XAI methods to trust our models - but can we trust XAI itself? Ironically, XAI comes with its own trust problem: these methods are approximations, with their own errors and pitfalls that are easy to overlook. Take the choice of dataset: should you compute your explanations on training or holdout (test) data? For loss-based feature importance methods like permutation feature importance, the answer is clear: use holdout data. Using training data will give you optimistically biased importance scores if the model overfits — a common pitfall.

I have asked myself the same question for years when it comes to feature effects like partial dependence plots (PDP) or accumulated local effects (ALE). Unfortunately, there are no clear recommendations on which dataset to use. Practitioners disagree, software defaults vary, and many researchers quietly sidestep the question. In this post, I want to shed light on the errors lurking in feature effects, and finally (!) give concrete recommendations on which dataset to use when computing them.

Feature effects: a quick refresher

Feature effects describe how changes in an input variable influence a model’s predictions. Take a concrete example: how is the predicted hourly power consumption in a city affected by the hour of the day?

Figure 1: Effects of the hour of the day on the power consumption prediction.

PDP and ALE give you exactly that: in our example (Figure 1), power consumption is lowest during night hours and peaks in the evening when people return from work.

PDPs work by marginalizing over all other features: for a given hour, you average predictions across all observed combinations of the remaining features. While simple and intuitive, this includes unrealistic combinations when features are correlated. ALE plots address this by working locally: they partition the feature into intervals, compute how predictions change within each interval, and accumulate those local effects. However, both are estimates, and estimates have errors.

Feature effects also have bias & variance

You’re probably familiar with the bias-variance tradeoff for models: a model’s mean-squared error (MSE) decomposes into a systematic bias and a variance component.

Figure 2: Bias (systematic deviation from the true value) vs. variance (random variability around it)

The same logic applies to feature effects: we can decompose a feature effect’s pointwise deviation from the “true underlying effect” in the data into now 4 components: bias and variance inherited from the model, and bias and variance introduced by the feature effect computation itself.

Model errors propagate to feature effects

Naturally, the errors of your machine learning model flow into your feature effects. If your model is biased (e.g., the complexity of your chosen model does not suffice to represent the highly nonlinear relationships in power consumption), this leads to a bias in your feature effect (e.g., your PDP of the feature “Hour” is systematically off).

Similarly, if your model has high variance, as can happen when you overfit to the training data, it affects the variance of your feature effects (e.g., you might get a very different PDP depending on the random seed used to train the model). We’ll come back to what you can do about this in a minute.

An additional bias on training data?

There is another bias component that now depends on how you compute the feature effects - and here is where the training-vs-holdout question becomes relevant. In theory, this bias component vanishes only when we compute feature effects on holdout data. However, we conducted a large simulation study, and the results indicated that the bias arising from training data is practically negligible. So, using training data seems to be safe, even when models overfit!

Size matters…

Similarly, there is another variance component, stemming from the fact that we compute feature effects on finite samples of data. Interestingly, this variance is connected to feature interactions learned by the model. For models without interactions (standard linear models, GAMs), it is zero. However, most machine learning models learn complex interaction patterns. In these cases, the variance decreases with increasing dataset size.

For PDPs, the scaling factor is 1/n. For ALE, it is K/n, where K is the number of intervals. This leads to two conclusions: (1) Dataset size matters, and since the training set is usually larger, we should prefer it for feature effect computation. (2) ALE is particularly sensitive to the dataset size, especially with fine-grained intervals.

Overfitting? Cross-validation helps

Earlier, I flagged that model variance from overfitting reflects in feature effect variance. So, what can you do if you suspect your model is overfitting?

The fix is cross-validation: fit multiple models on different folds, average the resulting feature effects, and you reduce model variance! At the same time, it preserves the effective sample size of training-data estimation (at the cost of extra computation).

Takeaways

1️⃣ XAI methods have errors - for feature effects, we can decompose them into bias and variance, stemming from the model error and feature effect computation itself.

2️⃣ Training data bias in feature effects appears negligible in practice - your feature effects won’t be meaningfully biased by using training instead of holdout data, even if your model overfits.

3️⃣ The training set is often preferable for feature effect computation - larger sample sizes considerably improve feature effects, particularly for ALE!

4️⃣ Overfitting models? Use cross-validation for feature effects - averaging feature effects across CV folds reduces model variance without sacrificing sample size.

We started with an uncomfortable truth: the tools we use to trust our models are themselves imperfect. Knowing where those imperfections lie doesn’t undermine the value of feature effects; it makes you a more honest user of them. And in this case, it comes with a practical finding: use your training data to estimate your feature effects - or cross-validation if your model overfits!

If you are interested in the details and theory: https://arxiv.org/pdf/2603.15057

This post explores the idea of context data and contrasts it with “classic” training data. Does moving from training data to context data change how we model? Does it enable something new?

Let’s dive in.

Training versus context

For traditional machine learning (linear regression, XGBoost, SVM), the training data shapes the model. Especially for trees, this is vivid: change the training data, and you might get out a differently shaped tree. Fit a linear regression model, and the weights (coefficients) become a function of the training data.

Not with tabular foundation models. Pre-trained. No classic training step. Prediction happens via in-context learning. When predicting with a TFM, you have to provide both the “training” (aka context) and the test data. Through multiple steps, the table cells are embedded, and through attention mechanisms and fully connected layers, the TFM enriches the cell representations by attending to other cells and computing stuff. All based on what the model learned in pre-training. Training and test data points can attend to the training data points, but not to other test data. The cell embeddings for y_test are used to predict or classify. If you want to learn more about how tabular foundation models work, check out my TabPFN architecture post.

So what happens when we change the context data? The model weights don’t change. The only things that change are the predictions that come out of the model: because the context changed, the embeddings will change, and the set of points that can be attended to will change.

But besides not affecting the model, is it really worth thinking about the “training” data as context data?

What I tell you in the next two sections is not new: You could also do traditional ML and call it context instead of training. Wrap it all in a function where the .predict() actually does fit+predict. Except, there are things that make it worthwhile to think of context data instead of training data: the ability and necessity to work with smaller datasets (which TFMs work well on) and the inverted cost in which training cost effectively disappears (ignoring pre-training here) and prediction becomes the expensive step.

Smaller and smarter contexts

Making predictions with tabular foundation models is relatively expensive. Especially if you always use all your training data as context. The inference runtime complexity of TabICL is O(n^2 + nm^2), where n is the number of rows of context+test and m is the number of columns. Since it’s quadratic in the number of rows, it will absolutely explode when we move up the orders of magnitude: a 10x increase in training data rows is a 100x increase in runtime (assuming that training data size is much larger than test data size and much larger than the number of features). The quadratic scaling strongly incentivizes using a smaller context dataset.

There is a more positive outlook on small contexts: Tabular foundation models work especially well for smaller data. That’s where they typically outperform most other approaches. Perhaps it’s because, for smaller data, the inductive biases learned in pre-training can shine.

Anyways, we can think of downsizing our data. When I experimented with tabular foundation models I sometimes downsampled the data for improved runtime, but with the drawback of reduced predictive performance.

For example, the following figure shows the mean absolute error for predicting wine quality, comparing TabICL and a random forest. The TabICL model has the same performance with 700 context data points as the random forest with 1200 training data points. However, increasing the context data for TabICL to the full 1200 data points gives us a big boost in MAE. 1200 data points is easily handled by a TFM nowadays. The question is whether, for example, downsampling from 1 million to 10k data points shows a relevant loss in predictive performance or not.

We can also be a bit smarter about downsizing the context:

• This paper suggests using a kNN model for each test data point to decide on the context. An interesting quote from the paper: “We thus believe that using nearby points as context is a good inductive bias for tabular data classification.”

• Another paper suggests clustering the training data. For a data point in the test data, check which cluster it belongs to, and then use the respective training cluster as context.

• You can reduce the training data to representative data points.

The following figure shows the cluster approach: Using 2 or 3 clusters is actually slightly more performant than using all data.

Again, all of these approaches are possible with the classic training-test scheme, but with tabular foundation models, we are more incentivized to reduce the context size while at the same time the TFM models work quite well on smaller data.

Context change is the new re-training

With in-context learning, the cost inverts between training and inference: training costs close to nothing while inference becomes more expensive.

This has implications for topics such as interpretability, feature selection, and robustness analysis. For example, in model-agnostic model interpretability, we have methods that rely on shuffling a feature column, but there is also a version of that which relies on removing that column and re-training the model. I’ve covered this in the following post:

In general, methods that require re-training are now much cheaper, relative to methods that require predicting twice with the same model. Usually, re-training was expensive. But for TFMs, re-training boils down to adapting the context data. The cost of re-training is close to zero, we only pay for making the predictions with TFMs.

Context engineering for TFMs?

So far, we have covered needs of smaller contexts for better scaling and changed costs for some post-hoc methods. Doesn’t feel like a big shift away from classic training+predict paradigm. I’m still trying to wrap my head around the context-data-framing. We gain a lot of flexibility without having to think about hyperparameter tuning, model selection, cross-validation, and so on. Just some random thoughts on what a context-paradigm might enable or simplify:

• Imagine a user requests their data to be fully deleted. Data your model was trained on. Removing them from a traditional ML model would require re-training, but for TFMs, it’s just a matter of removing their data from the context.

• Imagine a forecasting task for which only the last 30 days are relevant, because data distribution is drifting. No problem with TFMs, we can just have a sliding context window.

• Context data makes it easier to have per-request contexts. For example, contexts only with data from a certain region or customer segment, or even per user.

• If you have to remove poisoned/wrong/problematic data, just delete the rows from the context.

• Specialization is much easier. Imagine you have a global model that classifies transactions in your banking app. Then the user classifies some items on their own. These can be simply added to the context.

• Customization is also simplified. Continuing with the transactions example, the user could say: Don’t learn from old transactions (= remove rows from context), or don’t use the merchant name as a feature (= remove column from context).

None of this is truly new. But still, I found it very refreshing to think of context data rather than training data. And I feel like I have yet to fully grasp the context-mindset.

What if we always worked with machine learning models that produce the full predictive distribution? With classification, we are already at this point: Modern machine learning approaches output not just the majority class, but a probability for each class. Whether this probability is calibrated is another question.

With regression, we are a bit stuck with a point-based mindset. However, this could change with tabular foundation models. At least in theory: While these models produce the full predictive distribution (or at least a discretized approximation over a fixed support) it’s not the default and the output is a bit hidden.

Tabular foundation models are pre-trained to output the full predictive distribution

TabICL and TabPFN predict the full predictive distribution (discretized approximation). But by default, they only give you the conditional mean. Consider the following code snippet:

reg = TabPFNRegressor()
reg.fit(X=X_train, y=y_train)
predictions = reg.predict(X_test)

The predictions object in the code above contains just the means of the full predictive distributions. But in the background, the foundation model actually produces the full distribution (or at least a discretized version over a fixed support) and later aggregates it.

If you want the predictive distribution, you need to change a paramter in .predict():

quantiles_range = np.arange(0.05, 0.95, 0.05)
quantiles = reg.predict(X_test, output_type='quantiles', quantiles=quantiles_range)

Both output type options (mean, quantile) take the same amount of compute time, since the tabular foundation models predict the distribution anyways.

Why prefer distributions over points?

Starting from P(Y|X = x) may seem more tedious, but it gives us more modeling freedom. There are many situations where the mean is not appropriate, or where we want more informative predictions:

• You might prefer the predictive median over the mean to get more robust predictions.

• If you are interested in the tails of the distribution, you can extract the 10% and the 90% quantiles (or any other quantiles for that matter).

• Computing quantile intervals or the variance of the predictive distribution can help quantify uncertainty.

• Maybe you need the probability that the prediction exceeds a certain threshold, P(Y>threshold).

• Maybe you want to extract the modalities.

• Or you can also visualize the entire distribution.

We have so many more options when we work with the predictive distribution. And we get it without any additional computations when using tabular foundation models.

Are the predictive distributions calibrated?

While TFMs are directly pre-trained to predict the full distribution, there is no guarantee that this distribution is calibrated, meaning, e.g., that the predicted 10% quantile matches the actual 10% quantile. What distinguishes TFMs from approaches like linear regression or Bayesian regression models is the lack of explicit distributional assumptions.1 But to get a better idea whether to trust the full distributions, we need benchmarks. Unfortunately, popular benchmarks like TabArena focus on point estimates, not the entire distribution. This paper criticizes the field’s focus on the conditional mean and proposes reporting metrics in benchmarks that reflect calibration. Keep in mind that even when a model scores well in a benchmark, you have no guarantees for your own project. It’s just a rough pointer.

In the end, you need to validate calibration yourself, using proper scoring rules, or even calibrate using conformal prediction. But still, I find it exciting that tabular foundation models give us the option of predicting other aspects of the predictive distribution.

With the rise of tabular foundation models, a benchmark that often comes up is TabArena. Tabular, of course, is already way more mature than deep learning was in 2012. However, TabArena still introduces a new dynamic and is a change from other tabular benchmarks. For example, two days ago, Prior Labs announced TabPFN v2.6. A huge part of that announcement was the TabArena placement of the new model (first spot). TabArena has become a central element in the paradigm shift to tabular foundation models.

So what is TabArena? Let’s find out.

TabArena “lives” on HuggingFace

TabArena is a “living” benchmark, and you can find it in its natural habitat on HuggingFace. It’s by far not the only tabular benchmark; there have been many before it, like OpenML-CC18 or Tabzilla.

These benchmarks have been crucial in advancing machine learning. Tabular benchmarks, at their minimum, define a collection of ML tasks connected to datasets. What sets them apart from a pure dataset repository, like UCI, is stronger curation of datasets, and sometimes a benchmarking protocol, e.g., for how to evaluate models.

TabArena is probably the benchmark with the strictest protocol and thoroughness:

• TabArena standardized the pre-processing and evaluation procedures.

• It also includes ensembles for each model (except TabICL and TabDPT).

• Has the highest limit for tuning and training the model, meaning more confidence that the search is maxed out.

• Not only metric results, but also predictions are available.

What makes it “live” is that it’s an actively maintained benchmark. Not just a static table in a paper, but a space on HuggingFace that the TabArena maintainers update with new ML algorithms and tabular foundation models. TabArena was created by researchers from Amazon Web Services, the University of Freiburg, the University of Mannheim, INRIA Paris, Ecole Normale Supérieure, the ELLIS Institute Tübingen, and Prior Labs. The core maintainers are Nick Erickson, Lennart Purucker, Andrej Tschalzev, and David Holzmüller.

TabArena pits ML algorithms against each other

TabArena rates ML algorithms and foundation models using an Elo rating system, which you might know from chess or other competitive games. Elo is for pairwise comparisons, and the rating of an ML algorithm / TFM can be interpreted as the expected win probability for a task. Winning, as in producing a better model than another algorithm on a given task. The following figure shows the data going into the Elo ranking. For example, TabICLv2 was beating CatBoost in 76% of tasks, or, inversely, CatBoost was beating TabICLv2 in 24% of the TabArena tasks.

Elo reflects well how we select models in practice. When we compare models for a project, we want one that beats the others in terms of performance. There may be other constraints, such as interpretability and computational performance, but predictive performance ranks high. Elo only counts wins versus losses (or ties), but not by which margin. If two algorithms have the same win rate, their Elo will be the same, even when one algorithm fails catastrophically on the losses, and the other always comes in second by a small margin. This, however, is covered by other metrics on TabArena, such as improvability.

TabArena contains 51 tasks

TabArena benchmarks ML approaches based on 13 regression and 38 classification datasets.

Sounds small.

Especially small, given the amount of tabular data flying all around. To be fair, most of them are locked up in companies. You won’t find Aldi sales data, OpenAI churn tables, or spring coil test datasets flying around the internet with an open license. But still, OpenML hosts over 6k datasets! The only problem is: once you start being just a little selective, the number drops fast.

The TabArena team started with 1053 datasets and sequentially filtered them down:

• They removed 491 duplicate datasets.

• 135 datasets were tabular only in disguise: they were actually derived from other data modalities like images.

• The team removed another 123 for lack of a real predictive task (e.g., deterministic outcomes).

• Tiny data, quality issues, incompatible license … meant another 181 were excluded.

• Of the remaining data, they reduced 70 non-IID datasets.

Leaving only 51 datasets after this manual process, see also the TabArena paper.

TabArena snapshot

Let’s finally have a look at the current leaderboard of the overall benchmark:

Current ceiling: AutoGluon 4h extreme. AutoGluon is an open-source AutoML framework by AWS, and “4h extreme” is a preset that allows up to 4 hours of training time with the most intensive ensemble and hyperparameter search settings for maximum predictive performance. The top 3 positions are all tabular foundation models: TabPFN 2.6, RealTabPFN-2.5, and TabICLv2. All the way down at position 7 appears the first boosted tree algorithm with LightGBM. Reminder: Elo rankings don’t tell us about the absolute differences in performance. Also, the number of datasets is a caveat, and keep in mind that TabArena represents small to mid-sized IID data. Nonetheless, still impressive how far tabular foundation models have come.

If you want to dive deeper, there is a TabArena presentation on YouTube.

Tabular foundation models are a paradigm shift from training+predict to pretraining+ICL (in-context learning), making it a great test for model-agnostic interpretability tools.

Let’s dive in.

The remainder of this post is based on research from the paper Interpretable Machine Learning for TabPFN.1

Model-agnostic interpretation works out of the box for tabular foundation models

Consider this minimal workflow: You train a model, and then study its feature importance using permutation feature importance (PFI). PFI is a classic, post-hoc model-agnostic interpretability method. Meaning it works regardless of the underlying model structure. It only needs access to the .predict() function. Behind “.predict()” could even secretly be Bob from marketing filling that Excel based on gut feeling. If PFI works for Bob, it will work for TFMs, right? Indeed, it does work and looks and feels very familiar.

reg = TabICLRegressor()
reg.fit(X_train, y_train)
pfi = sklearn.inspection.permutation_importance(reg, X_test, y_test)

The sklearn implementation of PFI computes R-squared for the test data, then shuffles each feature and checks R-squared again. The larger the drop, the more important that feature was. This is repeated for all features, typically averaged over a couple of such permutations, five by default.

So far, so familiar.

If it weren’t for the changed costs.

Inference-cost inversion changes the economics of interpretability

On my MacBook Air M1, computing PFI took 104 seconds for this dataset of 1k rows (700 “training”, and 300 test data). Way more than a random forest would need.

The reason is that tabular foundation models are still slow. Not for training, but at prediction time. While traditional machine learning is training-expensive and inference-cheap, this relationship inverts for tabular foundation models: “Training” costs almost nothing, but predictions are expensive. Note that I’m excluding pre-training here, because I’m viewing it from an application perspective. Side note: While not a classic real training step, reg.fit() loads the TFM’s weights and pre-processes the “training” data, aka the context for in-context learning.

We could just ignore this cost inversion and throw more compute at our problem. Absence of training is neat anyway, and maybe we can just swallow the increased cost of inference. Certainly a possibility for the GPU-rich. But even then, the economics of interpretability are not so favorable. Permutation feature importance in the example above already requires 26x (1 + number of repetitions x number of features) times as many predictions as just predicting the test data. For an interpretability metric, you may only compute once. But if you start computing things like Shapley values along with every prediction, costs get out of hand quickly.

So what to do?

Better scaling for inference-heavy interpretability

We can speed up inference-heavy interpretability methods through TFM-friendly implementations. For traditionally trained models, it makes sense to chunk calls to the model: For PFI, you might call model.predict() with every new shuffling of the data. For TFMs, this no longer makes much sense. According to the paper “Interpretable Machine Learning for TabPFN”, the cost of inference per test data scales with O(ntrain^2/ntest), where ntrain is the size of the training data and ntest the size of the test data (in rows). So it’s better to bundle more test data into one call (if memory permits), because — relatively — it gets cheaper. I tested this out for computing PFI for one feature (computation time was averaged over 100x doing this). The naive version is two calls to the model, once with the original test data, once with the permuted. The batched version concatenates both before sending the data to the TFM for prediction.

Efficiency for other methods like ICE, ALE, and PDP can also benefit from such implementation changes.

Adapting interpretability tools for a training-cheap, inference-expensive world

LOCO importance is a model-agnostic interpretability method that can also compute feature importances. It stands for “leave-one-covariate-out” and relies on re-training: Remove a feature from the training data, re-train the model, and compare performances on test data.

With traditional machine learning, LOCO is expensive when compared to PFI. To compute LOCO importance for an xgboost model with 100 features, you have to re-train the model 100 times.

This relation changes with TFMs as they are training-cheap and interpretability methods based on re-training become more attractive. Because under TFMs, LOCO costs roughly the same as PFI, due to the lack of a training step. “Re-training” under the TFM paradigm just becomes another forward pass of the data with one column removed.

For the following figure, I computed PFI and LOCO for one feature for both TabICL and a random forest, repeated 100 times, and measured how long it takes:

The random forest is, in total, much cheaper; however, note that no hyperparameter tuning is involved, so it’s not a 100% fair comparison. Look at the relation between PFI computation time versus LOCO computation time. For TabICL, it’s roughly the same, while for the random forest, LOCO is 10x more expensive compared to PFI.

Note that LOCO and PFI define feature importance differently. With PFI, we stick to a fixed model and see how it adapts to shuffling data. With LOCO, we ultimately compare two different models. When features are correlated, LOCO and PFI importances diverge and have different interpretations. These differences boil down to the question of whether our interpretation is true to the data (LOCO) or true to the model (PFI). Or, in a more cynical view, yet another case of correlation ruining interpretability.

Other interpretability methods, like Shapley values, also have re-training-based versions, which become more attractive in conjunction with TFMs. But again, with a changed interpretation.

So what?

All model-agnostic tools are still available for tabular foundation models, but the costs have shifted. We can adapt interpretability methods to some degree, e.g., by batch calling, or we might shift to more training-focused versions.

And, who knows, we might also see different types of interpretability arise:

• Mechanistic interpretability for tabular foundation models.

• Model-specific implementations. Think of TreeSHAP for tree-based models ( PFNshap?)

• TFMs might enable new types of interpretability in the form of directly estimating properties of the underlying data.

I’m looking forward to seeing how interpretability for TFMs will evolve.

Then came the shipping container.

Containerization came with huge upfront costs: ships had to be refitted, ports needed new cranes, and processes had to adapt. On the upside, containers drastically reduced loading times and labor. It also standardized transportation of goods: The same container could be loaded onto a truck or train without unpacking. Containerization standardized how goods move. Once everything fit into the same container, the logistics became simpler and faster.

Machine learning has seen a similar kind of standardization. If you wanted to train an image recognition model for dogs in 2005, you’d maybe have applied edge detectors, thresholded intensities, and hand-crafted a feature vector. Then train something like a support vector machine on top. Every problem required its individual solution.

Today, you can start from the same place regardless of the task: a pre-trained foundation model. Take a Vision Transformer, fine-tune it on your data, or train a logistic classifier on the embeddings. And it will likely just work. Wherever you look, you’ll find pre-trained, often transformer-based, foundation models: for translation, image recognition, image segmentation, speech-to-text, text-to-speech, video generation, …

All data modalities and tasks are occupied by foundation models.

All? No! One small modality still holds out against them: tabular data.

But this resistance is crumbling.

TFMs are a fundamental shift, not just a performance trade-off

TabPFN opened the era of foundation models for tabular data. For small and mid-sized data, tabular foundation models now outperform other ML algorithms (see TabArena). I feel like the discussion in the ML community has since switched from “performance not there” to:

• “Is a slight increase in performance worth the increased compute?”

• “But it doesn’t work for large data!”

• “Wow, so much wasted compute when you could just use linear regression / random forest.”

I agree. For many applications, the increased compute time is an obstacle. But for how long will this be the case? Ultimately, I expect advances in hardware, architecture, and inference tricks to fix these concerns.

As a machine learning community, we are very much focused on benchmarks and computational performance. That’s both the strength and the weakness of machine learning. The focus on benchmarks was certainly difficult for me when I started publishing on ML interpretability, but that’s another story.

Purely judging tabular foundation models in terms of performance and compute distracts from the larger shift I believe tabular ML is undergoing. During my deep dive into tabular foundation models, my intuition grew that this is something big. It took me a while to figure out why exactly I think TFMs are a fundamental shift in data science. Why TFMs are not just slightly better-performing but more expensive models.

This post explains why I now believe TFMs are a really big deal.

PFNs are a universal engine for tabular ML

CatBoost, logistic regression, SVMs – the boxes, crates, and sacks of tabular ML – get the job done. Scikit-learn has done a great job of containerizing them at least API-wise with .fit, .predict, …. But underneath, they are very different algorithms, producing very different models. Some thrive on CPU, some on GPU. Very different optimization, failure modes, task-readiness, and so on. You always have to train from scratch, which is especially difficult with smaller datasets. Integrating prior domain knowledge is also inconsistent between ML algorithms. For example, if you suspect your data should have a maximum interaction depth of two, for linear models, you would have to actually add interaction terms, while for tree-based models, you would have to restrict the depth of learned trees.

All of this changes with tabular foundation models, specifically with prior-data fitted networks (PFNs). Pre-trained on millions of synthetic data examples, these transformer-based models are trained to perform in-context learning. Prediction essentially becomes a single forward pass of training and test data at once, and – without changing the weights – the model predicts the test data by attending to the training data.

To keep with the analogy of shipping, we finally have a general-purpose container for tabular machine learning, or even for data science more generally. The general purpose technology enabling this containerization is the prior-data fitted networks. Read my post about PFNs here. PFNs are a general method of essentially self-supervised learning that consists of creating synthetic (tabular) tasks, on which a (transformer-based) model is pre-trained to predict new data via in-context learning. Thanks to PFNs, we now have tabular foundation models that can do classification and regression without needing to be trained on the particular task at hand.

If classification and regression were the only applications, I probably wouldn’t call it a revolution. While these two tasks already make up a substantial portion of business and research tasks, tabular ML and data science are much larger. For TFMs, classification and regression were the first milestones to be taken seriously by the ML community.

But TFMs are not stopping there. Any data science task that you can express as a self-supervised learning problem (including a data generator) can be learned by the PFN “engine” to pre-train a TFM that then one-shots this task.

Let’s have a look at the implications.

If you can simulate it, you can pre-train for it

Off-the-shelf classification and regression TFMs already work for survival analysis and time series forecasting. That’s impressive and relevant, but not the paradigm shift I’m talking about. The big shift comes from being able to pre-train tabular foundation models for ANY data science problem for which we are able to simulate data with a ground truth. Researchers have already begun pre-training foundation models for other tasks:

• Time-series forecasting: TimePFN

• Anomaly Detection: FoMo-0D

• Classification and regression for graphs: GraphPFN.

• p>n prediction problems (like genomics): TabPFN-Wide

• Missing data imputation: TabImpute

• Clustering: TabClustPFN.

• Counterfactual fairness: FairPFN

• Estimate Shapley values: ExplainerPFN

• Causal inference: Do-PFN and CausalFM

• Bayesian Optimization surrogate model: PFNs4BO

This non-exhaustive list shows how versatile the PFN-engine is to pre-train tabular foundation models beyond classification and regression. Pre-training a TFM is not a small feat, especially since designing the prior, aka the data distribution, is not always easy. But once pre-trained, you can use it on any table for that same task. I expect this list to become even larger while also niching down into more specific use cases.

The PFN-engine also allows you to instill any inductive bias into the model that you can define via the prior. For example, you can robustify TFMs through adversarial training. Or, to absorb useful inductive biases from tree-based models, you can add tree-based relationships into the prior (see also this paper). This all comes on top of PFNs already being able to adapt to inductive biases through in-context learning alone, as research suggests.

And since they are so versatile and predict the entire posterior predictive distribution, not just a point estimate, TFMs can also be used for density estimation and synthetic data generation. Both are already built into TabPFN and would also be doable with other TFMs.

All these facts and developments show how much of a general-purpose technology PFNs are: There will be TFMs for every suitable data science problem soon.

Tabular ML now rides the deep learning wave

The non-tabular machine learning world has already converged on deep neural networks, and many even on transformer-based foundation models. Tabular is, or rather, was the last bastion to hold onto its boxes, crates, and sacks. The only modality not using the same “container” that the other modalities are using.

By switching over to tabular foundation models, we are essentially streamlining tabular with the other modalities. I liken technical developments to a river. The huge investments and talent in deep learning/transformers make this a very strong river. Tabular is now dipping into this flow, and the transformer river is carrying tabular with it. Suddenly, tabular ML also benefits from the vast ecosystem:

• Investments in GPU infrastructure now benefit tabular.

• Any further, more specific hardware for neural network training/inference makes tabular ML more efficient.

• Race to make LLM inference cheap also helps TFMs.

• Architectures (e.g., attention mechanisms) are getting better.

• All the optimizations and learnings for training and deploying LLMs, like LoRA, quantization, and Muon, may be transferred.

• We can build with very mature and user-friendly deep learning software like PyTorch (I still remember what a pain TensorFlow was in 2018).

• Tabular data now becomes more interesting to a large pool of AI investors.

All these developments carry tabular foundation models, but not SVMs or boosted trees. TFMs make tabular data science snap into the broader ecosystem.

Besides being carried, there’s improved interoperability with other modalities: TFMs give us embeddings for tabular data, making it straightforward to represent and reuse tables. For example, it has now become easier to train multi-modal models with tabular data as a modality.

I’m betting on foundation models

It’s been some time since I’ve been this excited about tabular ML. Tabular foundation models feel like the first refreshing change in a long time. I just don’t care about yet another algorithm that produces a tree ensemble.

Given my reasoning above (containerization), I’m personally betting on TFMs. (“Betting” as in investing my time and attention, not as in participating in prediction markets.) I am planning to learn more about them, to push them forward, and to educate. For example, I started working on TFM overview, a website that tracks libraries and labs.

Tabular foundation models (TFMs) are a paradigm shift from traditional tabular ML: They are transformer-based architectures pre-trained on synthetic data. There is no classic training step. Instead, TFMs predict the test data in a single forward pass of combined training and test data without any parameter updates (in-context learning).

These last few weeks of deep-dive have reshaped how I think about TFMs and tabular ML as a whole. I won’t claim I can predict the future. I’ve been completely wrong before, like about how good AI would become at coding. Instead of predictions, here are a few scenarios of increasing impact of TFMs (levels) on everyday tabular ML work.

Let’s dive in.

Level 1: TFMs enter the pantheon of ML algorithms

In most projects, you already try a couple of ML algorithms. Who cares that TabICL and other TFMs are pre-trained? Just one more algorithm to throw at your data. An expensive one, so maybe you just use it for smaller datasets. In this scenario, TFMs would become an established member of the ML algorithms that you find in benchmarks and that people try out in model selection.

Level 2: TFMs become the new Random Forest, aka the quick-and-dirty model

Whenever I encounter a new tabular prediction problem, I throw the random forest at it. Why the random forest? It’s reasonably fast and works well without hyperparameter tuning. For you, this may be a different algorithm that gives you that feeling of comfort and convenience. Maybe it’s linear/logistic regression or the support vector machine. In this scenario, TFMs would become the quick-and-dirty baseline for most supervised ML tasks. It would make sense: Great performance without hyperparameter tuning, implementations for both regression and classification, can handle missing data, can also predict quantiles, and so on.

Level 3: TFMs become synonymous with tabular ML

In this scenario, TFMs become the go-to approach for all supervised machine learning tasks, from classification and regression to survival analysis and time series forecasting. Kind of what LLMs did in the NLP space. TFMs would dominate all other ML algorithms in performance, at least for most cases, but also in convenience. TFMs would become the Swiss knife of tabular ML: they would work for all types of supervised ML tasks, handle missing data with ease, quantify uncertainty, generate new data if you want, and much more. Developers would build an entire ecosystem of tools around TFMs, from easy deployment to interpretability and monitoring. The ecosystem grows and becomes so powerful and convenient that it’s hard to switch back to any other ML algorithm.

Where do we stand? My impression

For Level 1, the barriers are cleared: The software is there, it’s usable, and there are enough clever tricks to make TFMs (somewhat) work even for larger datasets. I expect costs to keep dropping. However, it hasn’t permeated yet: Not everyone knows about TFMs, and it will take time for these models to reach all the toolboxes.

For me personally, Level 2 is cleared, at least for small datasets. TFMs will be my go-to model for small to mid-sized data. Too convenient not to. (Sorry, Random Forest.) Whether it’s true for others modelers will remain to be seen and depends on how computational costs evolve.

Level 3 is clearly not reached, and I am not sure if it ever will be. Will TFMs ever become the default for supervised ML? This strongly depends on how much improvements in TFM architectures and inference tricks can bring down computational costs. And this probably won’t be a global binary thing, but can also be quite different between communities. For example, the random forest has dominated ecology for years, while other communities haven’t embraced the Random Forest as much. Currently, there’s definitely an ecosystem growing around these TFMs. The TFM ecosystem could become a little modeling universe, covering all the supervised ML needs, so no reason to use anything outside this universe.

TFMs vs. traditional ML may become an irrelevant question

Developments are happening in parallel with agentic AI, or, more specifically, coding agents that may make this entire TFM discussion irrelevant. I’ve jumped on the vibe-coding bandwagon, coding an Android app for personal use (basically a recording app that dispatches my voice notes to the right places). I’ve been critical about vibe-coding, but my recent experience with Claude Code has drastically changed my view on agentic AI. I believe the world of software engineering has shifted quite a bit. Similar automation may be coming for data science and supervised machine learning (or kind of already is already here). This may make the entire TFM versus “traditional” ML discussion irrelevant, because these choices are abstracted away. At least from a user perspecte, as the data scientist / ML engineer (or whatever the job may be called then) operates on a higher level of abstraction.

This concludes my series on tabular foundation models. Which doesn’t mean that I won’t keep posting on it, it’s just giving me the freedom to write about other things as well.

This post walks through a TabICLv2 regression workflow: loading data, “fitting”, predicting, quantifying uncertainty, and interpreting the predictions with SHAP.

Let’s get started.

Preparing the dataset

We’ll use the Energy Efficiency dataset from UCI with 768 rows and 8 building features. The task is to predict heating and cooling loads (regression). First, let’s load the data and prepare it for modeling:

from ucimlrepo import fetch_ucirepo
import pandas as pd
from sklearn.model_selection import train_test_split

energy = fetch_ucirepo(id=242)
X = energy.data.features
y = energy.data.targets["Y1"]  # Heating Load

X.columns = ["Relative Compactness", "Surface Area", "Wall Area", "Roof Area",
    "Overall Height", "Orientation", "Glazing Area", "Glazing Area Distribution"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Train: {X_train.shape[0]} rows | Test: {X_test.shape[0]} rows")
X_train.head()

Fitting TabICLv2

Well, fitting isn’t the right word, but most tabular foundation models emulate the sklearn API. “Fitting” the TabICLv2 regression model means downloading ~110MB of model weights from Hugging Face on first use. After that, it’s cached locally. The fit step also loads the model weights and pre-processes the data, like standardizing the features.

Let’s have a look at how this works and how long it takes. For comparison, I also trained a random forest. By the way, I am doing everything on my MacBook Air M1, no GPU. Not ideal for tabular foundation models, a GPU would be faster.

import time
from tabicl import TabICLRegressor
from sklearn.ensemble import RandomForestRegressor

tabicl = TabICLRegressor(n_estimators=8, device="cpu", random_state=42)
rf = RandomForestRegressor(random_state=42)

for name, model in [("TabICL", tabicl), ("RandomForest", rf)]:
    t0 = time.time()
    model.fit(X_train, y_train)
    print(f"{name:<12} fit: {time.time() - t0:.2f}s")

TabICL fit: 0.28s

RandomForest fit: 0.08s

Both are quite fast. While the random forest was faster, the .fit() step is where traditional tabular ML is costly for larger datasets, while TFMs should still be fast since no training happens here.

Predicting with TabICLv2

Let’s make predictions with TabICLv2 and evaluate performance.

import numpy as np
from sklearn.metrics import root_mean_squared_error, mean_absolute_error
import matplotlib.pyplot as plt

results = {}
for name, model, kwargs in [
    ("TabICL",      tabicl, {"output_type": "mean"}),
    ("RandomForest", rf, {}),
]:
    t0 = time.time()
    y_pred = model.predict(X_test, **kwargs)
    results[name] = y_pred
    rmse = root_mean_squared_error(y_test, y_pred)
    mae = mean_absolute_error(y_test, y_pred)
    print(f"{name:<12}  RMSE={rmse:.3f}  MAE={mae:.3f}  predict={time.time()-t0:.2f}s")

TabICL RMSE=0.424 MAE=0.304 predict=2.84s

RandomForest RMSE=0.490 MAE=0.354 predict=0.01s

The Random Forest shows the usual pattern: Per data row, prediction is much cheaper, aka faster than training. For tabular foundation models, the predict step is the expensive one. And we can see it’s comparably slow. However, the out-of-the-box performance is much better than that of the random forest. This is by no means a real benchmark (e.g., no hyperparameter tuning). It shows, however, something I observed a few times now: You can just throw a TFM at these tabular tasks, and it just works.

Quantifying uncertainty with prediction intervals

Predicting the heating load is a regression task.

I find it interesting how the Soda lab at INRIA architected and pre-trained TabICLv2 for regression:

• TabICLv2 for regression is pre-trained separately from the classification model.

• TabICLv2 is pre-trained to predict 999 quantiles using the (aggregated) pinball loss, while TabPFN treats regression as bin classification.

• To predict the mean, the 999 quantile predictions are averaged.

• We can also extract any quantile: 5%, median, 95%, …

Since we get the entire predictive distribution, we can quantify uncertainty by outputting prediction intervals instead of point predictions:

quantiles = tabicl.predict(X_test, output_type="quantiles", alphas=[0.05, 0.95])
coverage = np.mean((y_test.values >= quantiles[:, 0]) & (y_test.values <= quantiles[:, 1]))
print(f"Empirical 90% coverage: {coverage:.1%}")

Empirical 90% coverage: 90.9%

A well-calibrated 90% interval should contain roughly 90% of true values. An empirical coverage of 90.9% is pretty good and well-calibrated. However, this is just one anecdote. I also tried TabICL on the watersupply competition I won: The performance, out-of-the-box, was slightly worse than my XGBoost ensemble (but still very good). However, the coverage of the prediction interval, which should have been 90%, was something like 60%. To be fair, I had the same problem with the XGBoost ensemble, which I fixed using conformal prediction. The same fix is available for TFMs.

Interpretability with SHAP

Tabular foundation models are not inherently interpretable models, given that they are deep transformer-based neural networks, meaning we have to use additional methods to interpret these models, such as Shapley values (ShAP):

import shap

X_test_sample = X_test.iloc[:30]

def predict_fn(x):
    return tabicl.predict(pd.DataFrame(x, columns=X_train.columns), output_type="mean")
 background=/__u/mindfulmodeler.substack.com/shap.sample(X_train, 50, random_state=42)
explainer = shap.PermutationExplainer(predict_fn, background)

# TabICL PermutationExplainer
t0 = time.time()
shap_values = explainer(X_test_sample)
print(f"SHAP completed in {time.time() - t0:.1f}s")

# RF TreeExplainer
t0 = time.time()
tree_explainer = shap.TreeExplainer(rf)
tree_explainer(X_test_sample)
print(f"TreeExplainer    (RF): {time.time() - t0:.3f}s")

# RF PermutationExplainer
t0 = time.time()
perm_explainer_rf = shap.PermutationExplainer(rf.predict, background)
perm_explainer_rf(X_test_sample)
print(f"PermutationExplainer (RF): {time.time() - t0:.1f}s")

shap.summary_plot(shap_values, X_test_sample, show=False)

SHAP completed in 2496.5s

TreeExplainer (RF): 0.139s

PermutationExplainer (RF): 7.8s

Ouch. Big ouch.

Especially since I’ve already severely dialed down the computational burden: I’ve reduced the background dataset to just 50 data points, and we are only computing the Shapley values for 30 of the 154 test data points. Again, it’s on a CPU, so a bit of a caveat for my anecdote here. But practically, it’s my current reality, as I’m used to running smaller data science projects on my MacBook.

The huge SHAP computation time gap between TabICL and the Random Forest exists because of two reasons:

• For TFMs, prediction is expensive compared to traditional tabular ML.

• Model-agnostic methods, especially Shapley values, require many model.predict() calls. The only reason why Shapley values are reasonable for traditional tabular ML is that we have a fast model-specific estimator for tree-based models, which dominate(d) tabular ML.

But not all is lost: In theory, with some adaptations, it should be possible to use something like the GradientExplainer for TFMs, a model-specific version for gradient-based models. I’ve also seen that TabPFN uses a different approach to estimating Shapley values by leveraging the in-context learning, as described in this paper (yet another paper in my ever-growing paper stack 🙃).

So I guess we will have to see some optimization taking place to use the good old model-agnostic interpretability methods, but hopefully they shall be usable again. At the same time, I expect tabular foundation models to become more efficient. Also, maybe I’ll have to start using a GPU 🤷‍♂️.