arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11137cs.LG
收藏

TuneTables:面向可扩展先验数据拟合网络的上下文优化

TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

New York University(纽约大学);University of Freiburg(弗莱堡大学);University of Maryland(马里兰大学)

Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, Colin White

AI总结:

本文提出TuneTables,一种通过上下文优化实现参数高效微调的先验数据拟合网络方法,将大数据集压缩为小规模可学习上下文,在98个数据集上平均性能超越CatBoost等19种算法,且仅需优化TabPFN不到5%的参数。

摘要:

尽管表格分类传统上依赖于从零开始训练,但最近一项名为先验数据拟合网络(PFNs)的突破对这一方法提出了挑战。与大型语言模型类似,PFNs利用预训练和上下文学习,在单次前向传播中就能在新任务上取得强大性能。然而,当前的PFNs存在一些限制,阻碍了其广泛采用。值得注意的是,TabPFN在小型表格数据集上表现非常出色,但其设计并不适用于对规模超过1000的数据集进行预测。在这项工作中,我们通过上下文优化克服了这些限制,并大幅提升了PFNs的性能。我们提出了TuneTables,这是一种面向PFNs的参数高效微调策略,可将大型数据集压缩为更小的可学习上下文。我们在98个数据集上对19种算法进行了广泛实验,发现TuneTables平均性能最佳,优于CatBoost等提升树方法,同时仅优化了TabPFN不到5%的参数。此外,我们还表明TuneTables可用作可解释性工具,甚至可以通过优化公平性目标来缓解偏差。我们在https://github.com/penfever/TuneTables开源了代码和原始结果。

英文摘要:

While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this approach. Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. However, current PFNs have limitations that prohibit their widespread adoption. Notably, TabPFN achieves very strong performance on small tabular datasets but is not designed to make predictions for datasets of size larger than 1000. In this work, we overcome these limitations and substantially improve the performance of PFNs via context optimization. We introduce TuneTables, a parameter-efficient fine-tuning strategy for PFNs that compresses large datasets into a smaller learned context. We conduct extensive experiments on 19 algorithms over 98 datasets and find that TuneTables achieves the best performance on average, outperforming boosted trees such as CatBoost, while optimizing fewer than 5% of TabPFN's parameters. Furthermore, we show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective. We open-source our code and raw results at https://github.com/penfever/TuneTables.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑