ElTab:弹性循环表格Transformer
ElTab: Elastically Looped Tabular Transformers
AI总结:
ElTab是一种弹性循环表格Transformer,通过弹性深度训练实现单一模型适配不同计算预算,在30个OpenML分类数据集上,其性能优于特定深度循环模型,推理成本低于TabPFN基线,实现了性能与成本的良好权衡。
摘要:
表格Transformer模型如TabPFN虽能实现出色的预测性能,但推理成本高昂,难以在资源受限场景中部署。循环Transformer通过权重共享减少参数数量,但提前停止会导致预测性能大幅下降。为解决该问题,我们提出弹性循环表格Transformer(ElTab),这是一种通过对随机采样的提前出口进行监督训练,可适配不同计算预算的单一模型。在30个OpenML分类数据集上,单个ElTab模型的性能与6种特定深度的循环模型相当,训练成本降低超一半;其推理浮点运算量(FLOPs)比12层TabPFN风格基线少25%,性能却与该基线相当。这些结果表明,弹性深度训练可生成能适配部署预算的单一模型,在预测性能、存储及计算成本间实现良好权衡。
英文摘要:
Tabular transformer models such as TabPFN achieve strong predictive performance but incur substantial inference costs, making them difficult to deploy in resource-constrained settings. While looped transformers reduce parameter count through weight sharing, they can lose significant predictive power when stopped early. To address this challenge, we introduce the Elastically Looped Tabular Transformer (ElTab), a single model trained to operate across different computational budgets through supervision at randomly sampled early exits. Across 30 OpenML classification datasets, a single ElTab model matches six depth-specific looped models while cutting training cost by more than half, and matches a 12-layer TabPFN-style baseline with 25% fewer inference FLOPs. These results show that elastic-depth training yields a single model that adapts to the deployment budget at hand, offering a favorable trade-off among predictive performance, storage, and computational cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。