TabDPT-Turbo：面向表格预测的高效上下文学习方法
TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction
浏览论文内容
中文总结 AI 辅助
本文提出TabDPT-Turbo模型，通过结合基于行的注意力、长上下文预训练及SSL预训练，在保持与TabDPT v1.1相当性能的同时大幅提升推理速度，是当前最快的表格预测基础模型。
中文摘要 AI 辅助
基于上下文学习的表格基础模型在质量和普及度上迅速提升，但近期采用基于单元架构或检索的方法，为追求原始性能牺牲了效率，限制了其在计算资源有限或推理速度关键场景的应用。本文采用替代方案，坚持基于行的注意力机制，同时结合长上下文预训练以消除对检索的需求，将其与架构改进及在新获取的更大真实数据集上的SSL预训练相结合，提出TabDPT-Turbo模型。该模型在TabArena-Lite、CC18和CTR23数据集上的默认性能与TabDPT v1.1相当，推理速度快几个数量级，是领先基础模型中整体最快的模型，已作为TabDPT v1.2发布。
英文摘要
Tabular foundation models, driven by in-context learning, have rapidly grown in quality and popularity. However, recent approaches with either cell-based architectures or retrieval have sacrificed efficiency for raw performance, restricting their utility in situations where compute is limited or inference speed is crucial. We adopt an alternate approach, sticking with row-based attention while incorporating long context pre-training to eliminate the need for retrieval. By combining this with architectural improvements and SSL pre-training on a newly-sourced, larger corpus of real data results, we present TabDPT-Turbo, a model that provides comparable default performance to TabDPT v1.1 on TabArena-Lite, CC18, and CTR23, at orders of magnitude faster. In our experiments, TabDPT-Turbo is the fastest model overall among leading foundation models. We have released the new model as TabDPT v1.2 at https://github.com/layer6ai-labs/TabDPT-inference.
