行动胜于言辞：用于生成式推荐的万亿参数序列转导器
Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
AI总结：
本文提出生成式推荐框架及 HSTU 架构，将推荐重构为序列转导任务，在 NDCG 和速度上显著超越基线，并展示万亿参数模型随计算量幂律扩展，已在线部署并提升指标 12.4%。
摘要：
大规模推荐系统的特征在于依赖高基数、异构特征，并且需要每天处理数百亿用户行为。尽管在包含数千个特征的海量数据上进行训练，工业界大多数深度学习推荐模型（DLRM）仍无法随计算规模扩展。受 Transformer 在语言和视觉领域成功的启发，我们重新审视推荐系统中的基础设计选择。我们在生成式建模框架（“生成式推荐器”）中将推荐问题重新表述为序列转导任务，并提出一种新架构 HSTU，专为高基数、非平稳流式推荐数据设计。HSTU 在合成数据集和公开数据集上的 NDCG 指标最多比基线高出 65.8%，在 8192 长度序列上比基于 FlashAttention2 的 Transformer 快 5.3 倍到 15.2 倍。基于 HSTU 的生成式推荐器具有 1.5 万亿参数，在在线 A/B 测试中将指标提升 12.4%，并已部署在拥有数十亿用户的大型互联网平台的多个界面上。更重要的是，生成式推荐器的模型质量在经验上随训练计算量呈幂律扩展，跨越三个数量级，直至 GPT-3/LLaMa-2 规模，这减少了未来模型开发所需的碳足迹，并进一步为推荐领域的首个基础模型铺平了道路。
英文摘要：
Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
