How the Rise of Tabular Foundation Models Is Reshaping Data Science

Deep learning has historically struggled to perform well on structured tabular data, despite its success with complex data like images and text. A new class of models, known as Tabular Foundation Models (TFMs), aims to solve this by applying principles from large language models. TFMs are pretrained on millions of synthetic tables, enabling them to perform "in-context learning" on new, unseen tabular datasets without requiring any retraining. This approach contrasts with traditional methods like XGBoost, which must be trained from scratch for each specific table. By using Transformer-based architectures, TFMs can learn a predictive model on-the-fly, potentially reshaping how data science is performed on structured data.