arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.06971cs.LG
收藏

使用 TabPFN 进行上下文数据蒸馏

In-Context Data Distillation with TabPFN

Layer 6 AI

Junwei Ma, Valentin Thomas, Guangwei Yu, Anthony Caterini

AI总结:

本文提出上下文数据蒸馏(ICD)方法,通过优化上下文突破 TabPFN 处理表格数据时的数据规模限制,使其在固定内存下能处理更大数据集,并在48个大型数据集上表现优异。

摘要:

基础模型已经彻底改变了计算机视觉和自然语言处理中的任务。然而,在表格数据领域,像 XGBoost 这样的基于树的模型继续占据主导地位。TabPFN 是一种专为表格数据定制的 transformer 模型,它反映了近期基础模型在卓越的上下文学习能力方面的特点,无需特定任务训练或超参数调优即可与 XGBoost 的性能相媲美。尽管前景广阔,但 TabPFN 的适用性受到其数据大小限制的阻碍,限制了其在实际场景中的使用。为了解决这个问题,我们提出了上下文数据蒸馏(ICD),这是一种通过优化 TabPFN 的上下文来有效消除这些限制的新方法。ICD 使 TabPFN 能够在固定内存预算下高效处理显著更大的数据集,改善了 TabPFN 的二次内存复杂度,但代价是线性数量的调优步骤。值得注意的是,在 OpenML 的 48 个大型表格数据集上,经过 ICD 增强的 TabPFN 相对于成熟的基于树的模型和现代深度学习方法展现出了非常强劲的性能。

英文摘要:

Foundation models have revolutionized tasks in computer vision and natural language processing. However, in the realm of tabular data, tree-based models like XGBoost continue to dominate. TabPFN, a transformer model tailored for tabular data, mirrors recent foundation models in its exceptional in-context learning capability, being competitive with XGBoost's performance without the need for task-specific training or hyperparameter tuning. Despite its promise, TabPFN's applicability is hindered by its data size constraint, limiting its use in real-world scenarios. To address this, we present in-context data distillation (ICD), a novel methodology that effectively eliminates these constraints by optimizing TabPFN's context. ICD efficiently enables TabPFN to handle significantly larger datasets with a fixed memory budget, improving TabPFN's quadratic memory complexity but at the cost of a linear number of tuning steps. Notably, TabPFN, enhanced with ICD, demonstrates very strong performance against established tree-based models and modern deep learning methods on 48 large tabular datasets from OpenML.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑