发表机构
Institute of Statistics and Big Data, Renmin University of China（中国人民大学统计与大数据研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出SOMTab架构，将表格上下文学习的表示构建与查询条件检索分离，结合DCH-TailMix合成先验，在接近Transformer模型性能的同时提升了效率。
AI 中文摘要
基于上下文学习的表格基础模型已成为特定任务模型拟合的有力替代方案，但当前性能前沿仍由全流程使用注意力的架构主导，这引出了表格上下文学习的每个阶段是否都需要注意力的问题。本文提出SOMTab，一种用于高效表格上下文学习的集合-顺序Mamba架构，它将表示构建与查询条件检索分离：针对行和列表示，将无序表格令牌映射为稳定的潜在槽并应用基于Mamba的状态空间混合以构建紧凑表示；针对最终预测，保留基于注意力的上下文学习以保留带标签上下文示例的查询条件检索。本文还提出DCH-TailMix，一种结合度校正图异质性与混合重尾机制的合成先验，以丰富合成依赖结构。在多个表格基准测试中，SOMTab的性能接近强大的基于Transformer的表格基础模型，同时实现了更快的推理速度和更低的GPU内存占用，取得了良好的效率-精度权衡。
英文摘要
Tabular foundation models based on in-context learning have recently emerged as strong alternatives to task-specific model fitting. However, the current performance frontier remains dominated by attention-heavy architectures, where attention is used throughout the modeling pipeline. This raises a natural question: is attention necessary at every stage of tabular in-context learning? We introduce SOMTab, a Set-Order Mamba architecture for efficient tabular in-context learning. SOMTab separates representation construction from query-conditioned retrieval. For row and column representations, it maps unordered table tokens into stable latent slots and applies Mamba-based state-space mixing to construct compact representations. For final prediction, it retains attention-based in-context learning to preserve query-conditioned retrieval from labeled context examples. We further introduce DCH-TailMix, a synthetic prior that combines degree-corrected graph heterogeneity with mixed heavy-tailed regimes to diversify synthetic dependency structures. Across tabular benchmarks, SOMTab approaches the performance of strong Transformer-based tabular foundation models while achieving faster inference and lower GPU memory usage, yielding a favorable efficiency--accuracy trade-off.
