arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.02545cs.LGcs.AI
收藏

Wukong:迈向大规模推荐的缩放定律

Wukong: Towards a Scaling Law for Large-Scale Recommendation

Meta AI
Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, Guna Lakshminarayanan, Ellie Dingqiao Wen, Jongs… 展开作者

Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, Guna Lakshminarayanan, Ellie Dingqiao Wen, Jongsoo Park, Maxim Naumov, Wenlin Chen

AI总结:

针对推荐模型缺乏缩放定律的问题,提出完全基于堆叠因子分解机的架构Wukong及协同扩展策略,在多个数据集上质量优于SOTA并实现跨越两个数量级的缩放定律。

摘要:

缩放定律在模型质量的持续提升中发挥着关键作用。遗憾的是,由于扩展机制的效率低下,目前的推荐模型并未表现出类似于大语言模型领域中所观察到的缩放定律。这一限制在使这些模型适应日益复杂的真实世界数据集时带来了重大挑战。在本文中,我们提出了一种完全基于堆叠因子分解机的有效网络架构,以及一种协同扩展策略,统称为Wukong,以在推荐领域建立缩放定律。Wukong的独特设计使其能够仅通过更高更宽的层来捕获多样的任意阶交互。我们在六个公开数据集上进行了广泛评估,结果表明Wukong在质量上始终优于最先进的模型。此外,我们在一个内部大规模数据集上评估了Wukong的可扩展性。结果显示,Wukong在质量上保持对最先进模型的优越性,同时在跨越两个数量级的模型复杂度中遵循缩放定律,扩展到超过100 GFLOP/example,而先前的方法在此表现不佳。

英文摘要:

Scaling laws play an instrumental role in the sustainable improvement in model quality. Unfortunately, recommendation models to date do not exhibit such laws similar to those observed in the domain of large language models, due to the inefficiencies of their upscaling mechanisms. This limitation poses significant challenges in adapting these models to increasingly more complex real-world datasets. In this paper, we propose an effective network architecture based purely on stacked factorization machines, and a synergistic upscaling strategy, collectively dubbed Wukong, to establish a scaling law in the domain of recommendation. Wukong's unique design makes it possible to capture diverse, any-order of interactions simply through taller and wider layers. We conducted extensive evaluations on six public datasets, and our results demonstrate that Wukong consistently outperforms state-of-the-art models quality-wise. Further, we assessed Wukong's scalability on an internal, large-scale dataset. The results show that Wukong retains its superiority in quality over state-of-the-art models, while holding the scaling law across two orders of magnitude in model complexity, extending beyond 100 GFLOP/example, where prior arts fall short.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑