arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.04573cs.LGcs.AI
收藏

用于表格预测任务的对抗预训练 Transformer 的零样本元学习

Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformer

University of California, Berkeley(加州大学伯克利分校)

Yulun Wu, Doron L. Bergman

AI总结:

本文提出 APT,通过对抗性合成数据智能体预训练 Transformer,并引入可处理任意类别数的混合块架构,实现表格任务零样本元学习,性能媲美先进方法且运行更快。

摘要:

我们提出了一种对抗预训练 Transformer(Adversarially Pre-trained Transformer,APT),它能够在表格预测任务上执行零样本元学习,而无需在任何真实世界数据集上进行预训练,这是对 Prior-Data Fitted Networks(PFN)和 TabPFN 最新进展的扩展。具体而言,APT 使用对抗性合成数据智能体进行预训练,这些智能体不断改变其底层数据生成分布,并有意识地用不同的合成数据集挑战模型。此外,我们提出了一种混合块架构,能够处理具有任意类别数量的分类任务,解决了类别数量限制这一以往深度表格零样本学习器的关键弱点。在实验中,我们表明,在不根据类别数量和缺失值数量等数据集特征进行过滤的情况下,我们的框架在小型分类任务上达到了最先进的性能,同时保持平均运行时间在一秒以内。在分类和回归的常见基准数据集套件上,我们表明对抗预训练能够增强 TabPFN 的性能。在分析中,我们证明对抗性合成数据智能体相比 TabPFN 中的普通随机生成器能够生成更加多样化的数据集合。此外,我们证明我们的混合块神经设计提高了泛化能力,并大幅加速了预训练。

英文摘要:

We present an Adversarially Pre-trained Transformer (APT) that is able to perform zero-shot meta-learning on tabular prediction tasks without pre-training on any real-world dataset, extending on the recent development of Prior-Data Fitted Networks (PFNs) and TabPFN. Specifically, APT is pre-trained with adversarial synthetic data agents, who continue to shift their underlying data generating distribution and deliberately challenge the model with different synthetic datasets. In addition, we propose a mixture block architecture that is able to handle classification tasks with arbitrary number of classes, addressing the class size limitation -- a crucial weakness of prior deep tabular zero-shot learners. In experiments, we show that our framework matches state-of-the-art performance on small classification tasks without filtering on dataset characteristics such as number of classes and number of missing values, while maintaining an average runtime under one second. On common benchmark dataset suites in both classification and regression, we show that adversarial pre-training was able to enhance TabPFN's performance. In our analysis, we demonstrate that the adversarial synthetic data agents were able to generate a more diverse collection of data compared to the ordinary random generator in TabPFN. In addition, we demonstrate that our mixture block neural design has improved generalizability and greatly accelerated pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑