arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.10016cs.IRcs.AI
收藏

Kunlun: 通过统一架构设计建立大规模推荐系统的缩放定律

Kunlun: Establishing Scaling Laws for Massive-Scale Recommendation Systems through Unified Architecture Design

Meta Platforms, Inc.(Meta平台公司);OpenAI
Bojian Hou, Xiaolong Liu, Xiaoyi Liu, Jiaqi Xu, Yasmine Badr, Mengyue Hang, Sudhanshu Chanpuriya, Junqing Zhou, Yuhang Yang, Han Xu, Qiuling Suo, Laming Chen, Y… 展开作者

Bojian Hou, Xiaolong Liu, Xiaoyi Liu, Jiaqi Xu, Yasmine Badr, Mengyue Hang, Sudhanshu Chanpuriya, Junqing Zhou, Yuhang Yang, Han Xu, Qiuling Suo, Laming Chen, Yuxi Hu, Jiasheng Zhang, Huaqing Xiong, Yuzhen Huang, Chao Chen, Yue Dong, Yi Yang, Shuo Chang, Xiaorui Gan, Wenlin Chen, Santanu Kolay, Darren Liu, Jade Nie, Chunzhi Yang, Ellie Wen, Jiyan Yang, Huayu Li

AI总结:

针对大规模推荐系统缺乏可预测缩放定律的问题,提出Kunlun架构,通过低层优化(GDPA、HSP、滑动窗口注意力)和高层创新(CompSkip、事件级个性化)提升模型效率,MFU从17%提升至37%,缩放效率翻倍,已在Meta广告模型部署。

摘要:

推导可预测的缩放定律,即模型性能与计算投入之间的关系,对于大规模推荐系统的设计和资源分配至关重要。虽然这类定律已在大型语言模型中建立,但在推荐系统中仍具挑战,尤其是处理用户历史记录和上下文特征的系统。我们识别出低缩放效率是可预测幂律缩放的主要障碍,源于低模型FLOPs利用率(MFU)的模块和次优的资源分配。我们引入Kunlun,一种可扩展的架构,系统性地提升模型效率和资源分配。我们的低层优化包括广义点积注意力(GDPA)、分层种子池化(HSP)和滑动窗口注意力。高层创新包括计算跳过(CompSkip)和事件级个性化。这些进步在NVIDIA B200 GPU上将MFU从17%提升至37%,并将缩放效率相比最先进方法提升一倍。Kunlun现已部署在主要的Meta广告模型中,产生显著的生产影响。

英文摘要:

Deriving predictable scaling laws that govern the relationship between model performance and computational investment is crucial for designing and allocating resources in massive-scale recommendation systems. While such laws are established for large language models, they remain challenging for recommendation systems, especially those processing both user history and context features. We identify poor scaling efficiency as the main barrier to predictable power-law scaling, stemming from inefficient modules with low Model FLOPs Utilization (MFU) and suboptimal resource allocation. We introduce Kunlun, a scalable architecture that systematically improves model efficiency and resource allocation. Our low-level optimizations include Generalized Dot-Product Attention (GDPA), Hierarchical Seed Pooling (HSP), and Sliding Window Attention. Our high-level innovations feature Computation Skip (CompSkip) and Event-level Personalization. These advances increase MFU from 17% to 37% on NVIDIA B200 GPUs and double scaling efficiency over state-of-the-art methods. Kunlun is now deployed in major Meta Ads models, delivering significant production impact.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑