Why In-Context Learning Transformers Are Tabular Data Classifiers

The recently introduced TabPFN pretrains an In-Context Learning (ICL)transformer on synthetic data to perform tabular data classification. Assynthetic data does not share features or labels with real-world data, theunderlying mechanism that contributes to the success of this method remainsunclear. This study provides an explanation by demonstrating thatICL-transformers acquire the ability to create complex decision boundariesduring pretraining. To validate our claim, we develop a novel forest datasetgenerator which creates datasets that are unrealistic, but have complexdecision boundaries. Our experiments confirm the effectiveness ofICL-transformers pretrained on this data. Furthermore, we create TabForestPFN,the ICL-transformer pretrained on both the original TabPFN synthetic datasetgenerator and our forest dataset generator. By fine-tuning this model, we reachthe current state-of-the-art on tabular data classification. Code is availableat https://github.com/FelixdenBreejen/TabForestPFN.