arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.07585cs.LGcs.AI
收藏

基于Transformer的序列推荐模型规模扩展

Scaling Sequential Recommendation Models with Transformers

Mercado Libre Inc.(美客多公司)

Pablo Zivic, Hernan Vazquez, Jorge Sanchez

AI总结:

本研究受大语言模型缩放定律启发,利用完整Amazon Product Data数据集探索Transformer在序列推荐中的缩放规律,验证了计算最优训练可行性与下游迁移效果,为真实高维推荐场景的模型训练部署提供路线图。

摘要:

用户偏好建模主要通过分析用户与系统中不同可用元素的交互历史来实现。基于历史数据为个体偏好定制内容是序列推荐的核心目标。\n该问题的特性,以及Transformer架构在多个领域展现出的优异性能,推动了其在该领域的应用——已有研究证明,当模型参数量同步增加时,Transformer能有效利用规模不断扩大的训练数据。这种规模扩展特性受到了广泛关注,因为它为更大规模模型的设计与训练提供了重要指导。\n受大语言模型训练中观察到的缩放定律启发,我们探索了序列推荐领域的类似规律。\n我们使用了完整的Amazon Product Data数据集(以往研究仅对其进行了部分探索),并揭示了与语言模型中发现的相似的缩放行为。计算最优训练是可行的,但需要仔细分析该应用场景特有的计算量与性能权衡。\n我们还证明,通过在更小的特定任务领域上微调更大的预训练模型,性能缩放可以迁移到下游任务中。我们的方法和发现为真实高维偏好空间中的模型训练与部署提供了战略路线图,有助于提升训练和推理效率。\n我们希望本文能弥合Transformer的潜力与真实推荐系统中高维序列推荐固有复杂性之间的差距。\n代码和模型可在https://github.com/mercadolibre/srt获取。

英文摘要:

Modeling user preferences has been mainly addressed by looking at users' interaction history with the different elements available in the system. Tailoring content to individual preferences based on historical data is the main goal of sequential recommendation. The nature of the problem, as well as the good performance observed across various domains, has motivated the use of the transformer architecture, which has proven effective in leveraging increasingly larger amounts of training data when accompanied by an increase in the number of model parameters. This scaling behavior has brought a great deal of attention, as it provides valuable guidance in the design and training of even larger models. Taking inspiration from the scaling laws observed in training large language models, we explore similar principles for sequential recommendation. We use the full Amazon Product Data dataset, which has only been partially explored in other studies, and reveal scaling behaviors similar to those found in language models. Compute-optimal training is possible but requires a careful analysis of the compute-performance trade-offs specific to the application. We also show that performance scaling translates to downstream tasks by fine-tuning larger pre-trained models on smaller task-specific domains. Our approach and findings provide a strategic roadmap for model training and deployment in real high-dimensional preference spaces, facilitating better training and inference efficiency. We hope this paper bridges the gap between the potential of transformers and the intrinsic complexities of high-dimensional sequential recommendation in real-world recommender systems. Code and models can be found at https://github.com/mercadolibre/srt

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑