arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07298cs.IRcs.AI
收藏

原则性合成数据使推荐系统中的LLM首次出现缩放定律

Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation

Meta

Benyu Zhang, Qiang Zhang, Jianpeng Cheng, Hong-You Chen, Qifei Wang, Wei Sun, Shen Li, Jia Li, Jiahao Wu, Qunshu Zhang, Neeraj Bhatia, Xiangjun Fan, Hong Yan

AI总结:

本文提出一种分层框架生成高质量合成数据,通过避免原始数据噪声,首次在推荐领域实现LLM的稳健幂律缩放,并显著提升下游排序任务性能。

摘要:

大型语言模型(LLM)代表了推荐系统的一个有前景的前沿,但其发展一直受到缺乏可预测缩放定律的阻碍,而缩放定律对于指导研究和优化资源分配至关重要。我们假设,这可能是由于先前持续预训练(CPT)工作中原始用户交互数据固有的噪声、偏差和不完整性所致。本文介绍了一种新颖的分层框架,用于生成高质量合成数据,通过为LLM创建精心策划的教学课程来规避此类问题。我们提供了强有力的直接证据,证明我们课程的有效性:在原则性合成数据上训练的标准序列模型在下游排序任务中显著优于(在SasRec的recall@100上提高+130%)在真实数据上训练的模型,展示了其在学习可泛化用户偏好模式方面的优越性。在此基础上,我们首次通过实验证明,在高质量、推荐特定数据上持续预训练的LLM存在稳健的幂律缩放。我们的实验揭示了跨多种合成数据模态的一致且可预测的困惑度降低。这些发现为在推荐领域可靠地缩放LLM能力建立了基础方法论,从而将研究重点从缓解数据缺陷转向利用高质量的结构化信息。

英文摘要:

Large Language Models (LLMs) represent a promising frontier for recommender systems, yet their development has been impeded by the absence of predictable scaling laws, which are crucial for guiding research and optimizing resource allocation. We hypothesize that this may be attributed to the inherent noise, bias, and incompleteness of raw user interaction data in prior continual pre-training (CPT) efforts. This paper introduces a novel, layered framework for generating high-quality synthetic data that circumvents such issues by creating a curated, pedagogical curriculum for the LLM. We provide powerful, direct evidence for the utility of our curriculum by showing that standard sequential models trained on our principled synthetic data significantly outperform ($+130\%$ on recall@100 for SasRec) models trained on real data in downstream ranking tasks, demonstrating its superiority for learning generalizable user preference patterns. Building on this, we empirically demonstrate, for the first time, robust power-law scaling for an LLM that is continually pre-trained on our high-quality, recommendation-specific data. Our experiments reveal consistent and predictable perplexity reduction across multiple synthetic data modalities. These findings establish a foundational methodology for reliable scaling LLM capabilities in the recommendation domain, thereby shifting the research focus from mitigating data deficiencies to leveraging high-quality, structured information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑