arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2602.06563cs.IR
收藏

TokenMixer-Large: 在工业推荐系统中扩展大型排序模型

TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders

ByteDance AML(字节跳动 AML);ByteDance(字节跳动)
Yuchen Jiang, Jie Zhu, Xintian Han, Hui Lu, Kunmin Bai, Mingyu Yang, Shikang Wu, Ruihao Zhang, Wenlin Zhao, Shipeng Bai, Sijin Zhou, Huizhi Yang, Tianyi Liu, We… 展开作者

Yuchen Jiang, Jie Zhu, Xintian Han, Hui Lu, Kunmin Bai, Mingyu Yang, Shikang Wu, Ruihao Zhang, Wenlin Zhao, Shipeng Bai, Sijin Zhou, Huizhi Yang, Tianyi Liu, Wenda Liu, Ziyan Gong, Haoran Ding, Zheng Chai, Deping Xie, Zhe Chen, Yuchao Zheng, Peng Xu

AI总结:

TokenMixer-Large通过引入混合与反向操作、层间残差和辅助损失,提升了工业推荐系统中大型排序模型的扩展能力,实现了参数规模的显著提升和性能的显著改善。

摘要:

尽管推荐模型的扩展定律已获得显著关注,但现有架构如Wukong、HiFormer和DHEN往往在设计上不够优化且硬件利用率低,限制了其实际扩展能力。我们之前的TokenMixer架构(在RankMixer论文中提出)通过用轻量级的token-mixing操作替代自注意力,解决了有效性和效率的问题;然而,它在更深的配置中面临关键瓶颈,包括不 optimal 的残差路径、消失的梯度、不完全的MoE稀疏化以及受限的扩展性。在本文中,我们提出了TokenMixer-Large,一种系统演化的架构,专为极端规模的推荐系统设计。通过引入混合与反向操作、层间残差和辅助损失,我们确保了即使在模型深度增加时也能稳定地传播梯度。此外,我们还集成了稀疏按token的MoE,以实现高效的参数扩展。TokenMixer-Large在在线流量和离线实验中成功将参数扩展到70亿和150亿。目前在字节跳动的多个场景中部署,TokenMixer-Large在离线和在线性能上均取得显著提升,为电子商务带来了订单量增加+1.66%、每人预览支付GMV增加+2.98%、广告的ADSS提升+2.0%以及直播的收入增长+1.4%。

英文摘要:

While scaling laws for recommendation models have gained significant traction, existing architectures such as Wukong, HiFormer and DHEN, often struggle with sub-optimal designs and hardware under-utilization, limiting their practical scalability. Our previous TokenMixer architecture (introduced in RankMixer paper) addressed effectiveness and efficiency by replacing self-attention with a ightweight token-mixing operator; however, it faced critical bottlenecks in deeper configurations, including sub-optimal residual paths, vanishing gradients, incomplete MoE sparsification and constrained scalability. In this paper, we propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation. By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation even as model depth increases. Furthermore, we incorporate a Sparse Per-token MoE to enable efficient parameter expansion. TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments, respectively. Currently deployed in multiple scenarios at ByteDance, TokenMixer-Large has achieved significant offline and online performance gains, delivering an increase of +1.66\% in orders and +2.98\% in per-capita preview payment GMV for e-commerce, improving ADSS by +2.0\% in advertising and achieving a +1.4\% revenue growth for live streaming.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑