arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.06343cs.AI
收藏

注意差距?对现实和合成先验的分布比较用于表格基础模型

Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Models

School of Geographical Sciences University of Bristol, UK(布里斯托尔大学地理科学学院);School of Engineering Mathematics and Technology University of Bristol, UK(布里斯托尔大学工程数学与技术学院);School of Computer Science University of Bristol, UK(布里斯托尔大学计算机科学学院)

Alex O. Davies, Telmo de Menezes e Silva Filho, Nirav Ajmeri

AI总结:

本文比较了现实和合成先验在表格基础模型中的分布差异,发现合成先验在真实表格空间中占据狭窄区域,且无法通过优化超参数关闭匹配差距,表明分布差异对模型性能有影响。

摘要:

表格基础模型是在三种类型的数据集上预训练的:来自基准库的精选数据集、大规模网络上抓取的表格,或从参数生成先验中采样的合成表格。尽管预训练数据对模型性能至关重要,但对其分布关系及对下游性能影响的认识有限。本文分析了三个典型数据集:T4代表网络抓取数据,TabFM来自Kaggle的精选表格,TabICL是唯一有公开参数的合成先验。通过聚合特征和判别器AUCs及k-NN覆盖度量进行比较,发现TabICL合成先验占据真实表格空间的狭窄区域,优化超参数无法关闭匹配差距,且精选和网络抓取数据在特征空间上可互换。令人惊讶的是,合成预训练数据与真实表格的分布差距对基于特征接近度的测度或TabICL自身内部表示无明显影响,表明覆盖真实数据分布并非TabICL泛化的主要驱动因素。

英文摘要:

Tabular foundation models are pre-trained on one of three classes of corpus: curated datasets drawn from benchmark repositories, tables harvested at scale from the web, or synthetic tables sampled from a parametric generative prior. Despite the centrality of pre-training data to model performance, little is known about how these corpora relate to one another in distribution, and the impact this has on downstream performance. In this work we take three canonical, archetypal datasets used to train tabular foundation models; the T4 dataset represents web-scraped corpora, the TabFM dataset curated tables from Kaggle, and the TabICL dataset as the only well-used synthetic prior with publicly available parameters. We characterise each corpus using aggregate features over whole tables, columns and correlations, and compare them using discriminator AUCs and k-NN coverage metrics. We find that the TabICL synthetic prior occupies a narrow region of the space of real tables, that this mismatch cannot be closed by optimising prior hyper-parameters across more than 86 thousand configurations, and that curated and web-scraped corpora are broadly interchangeable on a distributional level in feature space. Surprisingly, the distributional gap between synthetic pre-training data and real tables has a clearly detectable effect on performance under neither feature-based proximity measures or TabICL's own internal representations, suggesting that coverage of the real-data distribution is not the primary driver of TabICL's generalisation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑