arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04692cs.LG
收藏

工程回归无需真实数据训练:使用多数据集嵌入进行表格基础模型的领域适应

Engineering Regression Without Real-Data Training: Domain Adaptation for Tabular Foundation Models Using Multi-Dataset Embeddings

Massachusetts Institute of Technology(麻省理工学院)

Lyle Regenwetter, Rosen Yu, Cyril Picard, Faez Ahmed

AI总结:

本文提出一种无需真实数据训练的工程回归方法,通过多数据集嵌入技术提升基础模型在数据稀少领域的表现。

摘要:

工程应用中的预测建模长期以来由定制模型和小规模、孤立的表格数据集主导,限制了大规模学习方法的应用。尽管最近在表格基础模型方面取得了进展,但用于预训练的合成训练分布可能无法反映工程数据的统计结构,限制了向工程回归的转移。我们介绍了TREDBench,一个包含83个真实世界表格回归数据集的精心挑选的集合,这些数据集带有专家工程/非工程标签,并使用TabPFN 2.5的Dataset级嵌入来研究公共表示空间中的领域结构。我们发现,工程数据集在某种程度上可以与非工程数据集区分开来,而标准程序生成的数据集则高度可区分于工程数据集,揭示了显著的合成-真实领域差距。为了在不使用真实工程样本进行训练的情况下弥合这一差距,我们提出了一种嵌入引导的合成数据编纂方法:我们生成并识别“工程样式的”合成数据集,并仅使用所选合成任务对TabPFN 2.5进行持续预训练。在35个工程回归数据集上,这种仅合成的适应提高了预测精度和数据效率,优于TabPFN 2.5在29/35个数据集上,优于AutoGluon在27/35个数据集上,平均乘法数据效率提升分别为1.75倍和4.44倍。更广泛地说,我们的结果表明,原理性的合成数据编纂可以将程序生成器转换为领域相关的“数据引擎”,使基础模型在真实数据收集是主要瓶颈的科学和工业领域中提高性能。

英文摘要:

Predictive modeling in engineering applications has long been dominated by bespoke models and small, siloed tabular datasets, limiting the applicability of large-scale learning approaches. Despite recent progress in tabular foundation models, the resulting synthetic training distributions used for pre-training may not reflect the statistical structure of engineering data, limiting transfer to engineering regression. We introduce TREDBench, a curated collection of 83 real-world tabular regression datasets with expert engineering/non-engineering labels, and use TabPFN 2.5's dataset-level embedding to study domain structure in a common representation space. We find that engineering datasets are partially distinguishable from non-engineering datasets, while standard procedurally generated datasets are highly distinguishable from engineering datasets, revealing a substantial synthetic-real domain gap. To bridge this gap without training on real engineering samples, we propose an embedding-guided synthetic data curation method: we generate and identify "engineering-like" synthetic datasets, and perform continued pre-training of TabPFN 2.5 using only the selected synthetic tasks. Across 35 engineering regression datasets, this synthetic-only adaptation improves predictive accuracy and data efficiency, outperforming TabPFN 2.5 on 29/35 datasets and AutoGluon on 27/35, with mean multiplicative data-efficiency gains of 1.75x and 4.44x, respectively. More broadly, our results indicate that principled synthetic data curation can convert procedural generators into domain-relevant "data engines," enabling foundation models to improve in data-sparse scientific and industrial domains where real data collection is the primary bottleneck.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑