发表机构
Carnegie Mellon University; Mohamed bin Zayed University of Artificial Intelligence; Guangdong University of Technology; University of California, San Diego（卡内基梅隆大学; 穆罕默德·本·扎耶德人工智能大学; 广东工业大学; 加利福尼亚大学圣迭戈分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
TAFFY通过上下文多样性先验和任务条件循环Transformer，增强表格基础模型的上下文学习能力，在多个基准上取得最低平均排名。
AI 中文摘要
表格基础模型的最新进展表明，在合成任务上进行训练可以显著提升上下文学习能力，其整体性能在很大程度上取决于模型在推理过程中能否从可用上下文中推断出特定任务的预测关系。在本文中，我们介绍了TAFFY，一种具有上下文多样性先验和任务条件循环Transformer的表格基础模型，以增强这一能力。具体而言，为了构建每个合成预训练上下文，上下文多样性先验从通过受控干预和分布偏移在共享因果过程上推导出的多个相关环境中进行采样。这种上下文多样性鼓励模型学习更全面且特定于任务的表示。此外，任务条件循环Transformer迭代地、选择性地应用一组共享的Transformer块来细化上下文表示，并通过任务条件门控调节最终隐藏状态的更新。这实现了任务自适应的迭代细化。这些组件共同鼓励模型在预训练期间从上下文对比中识别预测关系，并动态地为每个任务调节上下文整合。在六个分类和五个回归基准数据集上，TAFFY取得了最低的平均排名。
英文摘要
Recent progress in tabular foundation models suggests that training on synthetic tasks can substantially improve in-context learning capabilities, with overall performance largely depending on how well models can infer task-specific predictive relationships from the available context during inference. In this paper, we introduce TAFFY, a tabular foundation model with an In-Context Diversity Prior and a Task-Conditioned Looped Transformer that strengthen this ability. Specifically, to construct each synthetic pretraining context, the In-Context Diversity Prior samples from multiple related environments derived via controlled interventions and distribution shifts on a shared causal process. This in-context diversity encourages the model to learn a more comprehensive and task-specific representation. Moreover, the Task-Conditioned Looped Transformer iteratively and selectively applies a shared group of Transformer blocks to refine contextual representations, with a task-conditioned gate modulating the final hidden-state update. This enables task-adaptive iterative refinement. Together, these components encourage the model to identify predictive relationships from contextual contrasts during pretraining and dynamically modulate context integration for each task. Across six classification and five regression benchmark datasets, TAFFY attains the lowest average rank.
