arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16986cs.IRcs.SI
收藏

在大规模推荐系统中弯曲缩放定律曲线

Bending the Scaling Law Curve in Large-Scale Recommendation Systems

Meta Recommendation Systems(Meta推荐系统)
Qin Ding, Kevin Course, Linjian Ma, Jianhui Sun, Ruochen Liu, Zhao Zhu, Chunxing Yin, Wei Li, Dai Li, Yu Shi, Xuan Cao, Ze Yang, Han Li, Xing Liu, Bi Xue, Hongw… 展开作者

Qin Ding, Kevin Course, Linjian Ma, Jianhui Sun, Ruochen Liu, Zhao Zhu, Chunxing Yin, Wei Li, Dai Li, Yu Shi, Xuan Cao, Ze Yang, Han Li, Xing Liu, Bi Xue, Hongwei Li, Rui Jian, Daisy Shi He, Jing Qian, Matt Ma, Qunshu Zhang, Rui Li

AI总结:

ULTRA-HSTU通过创新的序列设计和稀疏注意机制,在大规模推荐系统中实现了显著的训练和推理效率提升,同时保持高质量的推荐性能。

摘要:

通过序列模型学习用户交互历史已成为大规模推荐系统的核心。近年来大型语言模型的进步揭示了有希望的缩放定律,引发了对长序列建模和更深层次架构的大量研究。然而,许多近期方法严重依赖交叉注意机制来解决序列建模中的二次计算瓶颈,这可能会限制自注意机制所能获得的表示能力。我们提出了ULTRA-HSTU,一种通过端到端模型和系统联合设计开发的新序列推荐模型。通过在输入序列设计、稀疏注意机制和模型拓扑结构上进行创新,ULTRA-HSTU在模型质量和效率上均取得了显著提升。全面的基准测试显示,ULTRA-HSTU在训练缩放和推理缩放方面分别比传统模型快5倍和21倍,同时提供更优的推荐质量。我们的解决方案已大规模部署,每日服务数十亿用户,并在实际生产环境中带来了显著的4%至8%的消费和参与度提升。

英文摘要:

Learning from user interaction history through sequential models has become a cornerstone of large-scale recommender systems. Recent advances in large language models have revealed promising scaling laws, sparking a surge of research into long-sequence modeling and deeper architectures for recommendation tasks. However, many recent approaches rely heavily on cross-attention mechanisms to address the quadratic computational bottleneck in sequential modeling, which can limit the representational power gained from self-attention. We present ULTRA-HSTU, a novel sequential recommendation model developed through end-to-end model and system co-design. By innovating in the design of input sequences, sparse attention mechanisms, and model topology, ULTRA-HSTU achieves substantial improvements in both model quality and efficiency. Comprehensive benchmarking demonstrates that ULTRA-HSTU achieves remarkable scaling efficiency gains -- over 5x faster training scaling and 21x faster inference scaling compared to conventional models -- while delivering superior recommendation quality. Our solution is fully deployed at scale, serving billions of users daily and driving significant 4% to 8% consumption and engagement improvements in real-world production environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑