TabularMath: 通过程序验证合成评估表格学习中的计算外推
TabularMath: Evaluating Computational Extrapolation in Tabular Learning via Program-Verified Synthesis
AI总结:
TabularMath通过程序验证合成评估表格学习中的计算外推能力,发现模型在统计插值与精确计算之间存在显著差异。
摘要:
标准表格基准主要关注模型在数据流形内部插值值的能力,其中擅长进行局部统计平滑的模型会受到奖励。然而,存在一个非常大的高价值表格数据类别,包括金融建模和物理模拟,这些数据是基于确定性计算过程生成的,而不是随机和噪声关系。因此,我们研究表格模型是否能从统计插值扩展到计算外推。我们提出了TabularMath,一个由基于GSM8K和AIME的验证程序生成的114个确定性问题(233,472行)的诊断基准。我们评估了9种表格架构和基于GPT-OSS-120B的上下文学习(ICL)。在标准回归度量上,TabPFN v2.5表现显著优异,达到R²=0.998的分布内表现,并在分布偏移下保持正R²,这在我们测试的表格模型中是唯一的。当我们测量四舍五入一致性(精确整数匹配)时,出现了不同的情况:TabPFN v2.5在分布外数据上低于10%,而ICL在分布外数据上保持在40%左右。R²与精确匹配准确率之间的差距表明,表格模型学习了平滑函数近似,但在外推时难以恢复精确的计算输出。这两种范式似乎互补:TabPFN能够高效地扩展数据;ICL从少量示例中实现精确计算。我们释放所有代码和数据以支持进一步研究。
英文摘要:
Standard tabular benchmarks mainly focus on the evaluation of a model's capability to interpolate values inside a data manifold, where models good at performing local statistical smoothing are rewarded. However, there exists a very large category of high-value tabular data, including financial modeling and physical simulations, which are generated based upon deterministic computational processes, as opposed to stochastic and noisy relationships. Therefore, we investigate if tabular models can provide an extension from statistical interpolation to computational extrapolation. We propose TabularMath, a diagnostic benchmark of 114 deterministic problems (233,472 rows) generated from verified programs based on GSM8K and AIME. We evaluate 9 tabular architectures and in-context learning (ICL) with GPT-OSS-120B. On standard regression metrics, TabPFN v2.5 performs remarkably well, achieving R^2=0.998 in-distribution and maintaining positive R^2 even under distribution shift, which is unique among the tabular models we tested. When we measure rounded consistency (exact integer match), a different picture emerges: TabPFN v2.5 drops below 10% on out-of-distribution data, while ICL maintains around 40%. This gap between R^2 and exact-match accuracy suggests that tabular models learn smooth function approximations but struggle to recover precise computational outputs under extrapolation. The two paradigms appear complementary: TabPFN scales efficiently with data; ICL achieves exact computation from few examples. We release all code and data to support further investigation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。