RankUp:面向大规模广告推荐系统的高秩表示
RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
AI总结:
本文提出RankUp架构,通过随机排列拆分稀疏特征、多嵌入范式、全局token整合和交叉预训练嵌入token,缓解表示崩溃并提升表达能力,在微信视频号、公众号和朋友圈等大规模生产环境中实现GMV提升。
摘要:
推荐系统中的扩展定律已被越来越多地验证,其中基于MetaFormer的架构一致受益于增加的模型深度、隐藏维度性和用户行为序列长度。然而,表示能力是否与参数增长成比例扩展仍不清楚。先前对RankMixer的研究表明,token表示的有效秩在层之间表现出阻尼振荡轨迹,未能随着深度一致增加,甚至在更深的层中退化。受此启发,我们提出了RankUp架构,通过随机排列拆分稀疏特征、多嵌入范式、全局token整合和交叉预训练嵌入token,以缓解表示崩溃并增强表达能力。RankUp已全面部署于微信视频号、公众号和朋友圈等大规模生产环境中,分别实现了3.41%、4.81%和2.12%的GMV提升。
英文摘要:
The scaling laws for recommender systems have been increasingly validated, where MetaFormer-based architectures consistently benefit from increased model depth, hidden dimensionality, and user behavior sequence length. However, whether representation capacity scales proportionally with parameter growth remains unexplored. Prior studies on RankMixer reveal that the effective rank of token representations exhibits a damped oscillatory trajectory across layers, failing to increase consistently with depth and even degrading in deeper layers. Motivated by this observation, we propose RankUp, an architecture designed to mitigate representation collapse and enhance expressive capacity through randomized permutation splitting over sparse features, a multi-embedding paradigm, global token integration and crossed pretrained embedding tokens. RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41%, 4.81% and 2.12%, respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。