Xiaomi-TabLDM：表格基础模型技术报告

中文摘要 AI 辅助

我们提出Xiaomi-TabLDM，这是一种通过上下文学习实现分类和回归任务的表格型大基础模型，无需针对特定任务进行微调即可提供卓越的预测精度。该模型仅在由结构因果模型（SCM）生成的合成数据上进行预训练，支持更灵活的上下文利用和更高效的容量扩展。i）新的性能标准：在基准测试中展现出强大的回归性能——Xiaomi-TabLDM在OpenML-CTR23上排名第1，在TALENT、TabArena和BCCO的回归任务中排名第2，在四个互补的基准套件中展现出一致的强回归性能；同时具备优异的性能-效率权衡，例如在TabArena回归任务中，它获得第二高的Elo值，且训练时间比排名第一的TabFM少82%，预测时间少68%。ii）大规模合成预训练：Xiaomi-TabLDM扩大了预训练所用合成表格数据的覆盖范围和多样性，还采用了三阶段训练策略，结合双流特征分组、轻量注意力残差以及稀疏混合专家（sparse Mixture-of-Experts），使其能够在多样化表格任务中学习更丰富的特征交互和专家专业化能力。iii）测试时扩展：Xiaomi-TabLDM通过测试时计算扩展进一步提升表格预测性能，即在推理时分配额外计算资源，持续提升相对于基础模型的预测性能。

英文摘要

We introduce Xiaomi-TabLDM, a tabular large data foundation model for classification and regression via in-context learning, which delivers superior prediction accuracy without requiring task-specific fine-tuning. Pretrained exclusively on synthetic data generated from structural causal models (SCMs), our model enables more flexible context utilization and more efficient capacity scaling. i) A new performance standard. Strong regression performance across benchmarks: Xiaomi-TabLDM ranks 1st on OpenML-CTR23 and 2nd on regression across TALENT, TabArena, and BCCO, demonstrating consistently strong regression performance across four complementary benchmark suites. Favorable performance--efficiency trade-off: Xiaomi-TabLDM combines strong predictive performance with substantially lower computational cost. For example, on TabArena regression, it achieves the second-highest Elo while using 82% less training time and 68% less prediction time than the top-ranked TabFM. ii) Large-scale synthetic pretraining. Xiaomi-TabLDM expands the coverage and diversity of synthetic tabular data used for pretraining. We also adopt a three-stage training strategy together with dual-stream feature grouping, lightweight Attention Residual, and sparse Mixture-of-Experts, enabling Xiaomi-TabLDM to learn richer feature interactions and expert specialization across diverse tabular tasks. iii) Test-time scaling. Xiaomi-TabLDM further extends tabular prediction through test-time compute scaling, where allocating additional computation at inference time consistently improves predictive performance over the base model.

发表机构

• Xiaomi（小米）

机构由 AI 辅助整理，请以论文原文为准。