arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.28554cs.LG
收藏

高性能,低可靠性:表格基础模型的不确定性基准测试

High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

CentraleSupélec(中央理工大学);ENS Paris-Saclay(巴黎-萨克雷大学);Université Paris-Saclay(巴黎-萨克雷大学)

José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, Bich-Liên Doan

AI总结:

通过TALENT基准测试,发现表格基础模型虽在预测性能上优于梯度提升决策树,但在不确定性校准上表现更差,存在性能-不确定性权衡。

摘要:

最近的表格基础模型(TFMs)展示了最先进的预测性能,通常超越梯度提升决策树(GBDTs)。然而,这些模型的可信度,特别是其不确定性量化,在很大程度上被忽视了。我们通过在TALENT基准测试的112个数据集上进行广泛研究,比较TFMs、GBDTs和经典基线,调查了这一差距。我们的结果揭示了性能-不确定性权衡:尽管TFMs在AUC测量下达到了最高的预测性能,但在共形预测下,它们表现出较低的条件覆盖率(由SSCS测量),相比GBDTs。在合成数据集上的补充实验进一步刻画了这种效应加剧的情景。我们得出结论,尽管TFMs推进了预测前沿,但实现良好校准的不确定性仍然是其可靠采用的主要开放挑战。代码可在:https://github.com/jose-melo/high-performance-low-reliability 获取。

英文摘要:

Recent Tabular Foundation Models (TFMs) have demonstrated state-of-the-art predictive performance, often surpassing Gradient-Boosted Decision Trees (GBDTs). However, the trustworthiness of these models, particularly their uncertainty quantification, has been largely overlooked. We investigate this gap through an extensive study comparing TFMs, GBDTs, and classical baselines on the 112 datasets of the TALENT benchmark. Our results reveal a performance-uncertainty trade-off: although TFMs achieve the highest predictive performance, measured by AUC, they exhibit lower conditional coverage under conformal prediction, measured by SSCS, compared to GBDTs. Complementary experiments on synthetic datasets further characterize the regimes in which this effect intensifies. We conclude that while TFMs advance predictive frontiers, achieving well-calibrated uncertainty remains a major open challenge for their reliable adoption. Code is available at: https://github.com/jose-melo/high-performance-low-reliability

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑