Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models
School of Geographical Sciences University of Bristol, UK(布里斯托尔大学地理科学学院);School of Engineering Mathematics and Technology University of Bristol, UK(布里斯托尔大学工程数学与技术学院);School of Computer Science University of Bristol, UK(布里斯托尔大学计算机科学学院)
Alex O. Davies, Telmo de Menezes e Silva Filho, Nirav Ajmeri
Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior. Despite the centrality of pre-training data to model performance, little is known about how these corpora relate to one another in distribution, and the impact this has on downstream performance. In this work we take three canonical, archetypal datasets used to train tabular foundation models; the T4 dataset represents web-scraped corpora, the TabFM dataset curated tables from Kaggle, and the TabICL dataset as the only well-used synthetic prior with publicly available parameters. We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics. We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior hyper-parameters across more than 86 thousand configurations, and that curated and web-scraped corpora are broadly interchangeable on a distributional level in feature space. Surprisingly, the distributional gap between synthetic pre-training data and real tables has a clearly detectable effect on performance under neither feature-based proximity measures or TabICL's own internal representations, suggesting that coverage of the real-data distribution is not the primary driver of TabICL's generalisation.
DenoGrad: A Gradient-Based Framework for Data Refinement in Tabular and Time-Series Learning
Department of Computer Science and Artificial Intelligence, University of Granada(格拉纳达大学计算机科学与人工智能系);Andalusian Institute of Data Science and Computational Intelligence (DaSCI), University of Granada(安达卢西亚数据科学与计算智能研究所(DaSCI),格拉纳达大学)
Shifts 2.0: Extending The Dataset of Real Distributional Shifts
University of Cambridge(剑桥大学);DeepSea;University of Basel(巴塞尔大学);University of Lausanne(洛桑大学);University of Applied Sciences of Western Switzerland(瑞士西部应用科学大学);Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院)
Limited Reference, Reliable Generation: A Two-Component Framework for Tabular Data Generation in Low-Data Regimes
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Financial Technology, Fudan University(复旦大学金融技术研究院);Northwestern Polytechnical University(西北工业大学)
Self-Improving Tabular Language Models via Iterative Reward-Guided Post-Training
Department of Engineering, University of Cambridge(剑桥大学工程系);CISPA Helmholtz Center for Information Security, Saarbrücken, Germany(德国萨尔布吕肯信息安全中心);The Alan Turing Institute, London(伦敦阿兰·图灵研究所)
MammoClean: Toward Reproducible and Bias-Aware AI in Mammography through Dataset Harmonization
Department of Mathematics and Statistics, Qatar University(数学与统计学系,卡塔尔大学);Department of Radiology, Northwestern University(放射学系,西北大学);Graduate School of Information Science, University of Hyogo(信息科学研究生院,广岛大学);Advanced Medical Engineering Research Institute, University of Hyogo(先进医学工程研究所,广岛大学)
Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN
University of Bergen Centre for the Science of Learning \& Technology (SLATE) Bergen Norway;Department of Computer Science University of Oxford;Alan Turing Institute Oxford United Kingdom;University of Bergen;University of Oxford;Alan Turing Institute