TabPFN-Wide:为极端特征数量继续预训练
TabPFN-Wide: Continued Pre-Training for Extreme Feature Counts
AI总结:
本文提出TabPFN-Wide模型,通过继续预训练合成数据提升对高维数据的处理能力,实现对超过3万特征的稳健建模,同时保持可解释性。
摘要:
揭示分子测量与病理学之间的关系仍是生物医学中机器学习的重要应用。该领域数据通常包含少量观测但数千个潜在噪声特征,对传统表格机器学习方法构成挑战。尽管先验数据拟合网络成为预测表格数据任务的基础模型,但它们目前无法处理大量特征(>500)。尽管特征降维可使其应用,但会阻碍特征重要性分析。我们提出了一种策略,通过在定制先验上继续预训练现有模型来扩展模型。所得到的模型TabPFN-Wide在匹配或超越基础模型性能的同时,表现出对噪声的更强鲁棒性。它能够无缝扩展到超过3万的分类和连续特征,无论噪声水平如何,同时保持内在可解释性,这对于生物医学应用至关重要。我们的结果表明,先验信息适应有助于增强基础模型在高维数据中的能力。在真实世界组学数据集上,我们显示模型识别出的许多最相关特征与先前的生物发现重叠,而其他特征则提出了未来研究的潜在起点。
英文摘要:
Revealing novel insights from the relationship between molecular measurements and pathology remains a very impactful application of machine learning in biomedicine. Data in this domain typically contain only a few observations but thousands of potentially noisy features, posing challenges for conventional tabular machine learning approaches. While prior-data fitted networks emerge as foundation models for predictive tabular data tasks, they are currently not suited to handle large feature counts (>500). Although feature reduction enables their application, it hinders feature importance analysis. We propose a strategy that extends existing models through continued pre-training on synthetic data sampled from a customized prior. The resulting model, TabPFN-Wide, matches or exceeds its base model's performance, while exhibiting improved robustness to noise. It seamlessly scales beyond 30,000 categorical and continuous features, regardless of noise levels, while maintaining inherent interpretability, which is critical for biomedical applications. Our results demonstrate that prior-informed adaptation is suitable to enhance the capability of foundation models for high-dimensional data. On real-world omics datasets, we show that many of the most relevant features identified by the model overlap with previous biological findings, while others propose potential starting points for future studies.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。