arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2111.11294cs.IRcs.LG
收藏

推荐模型的缩放定律:面向通用用户表示

Scaling Law for Recommendation Models: Towards General-purpose User Representations

NAVER CLOVA;NAVER AI Lab(NAVER人工智能实验室)

Kyuyong Shin, Hanock Kwak, Su Young Kim, Max Nihlen Ramstrom, Jisu Jeong, Jung-Woo Ha, Kyung-Min Kim

摘要:

大规模预训练模型的最新进展,如BERT、GPT-3、CLIP和Gopher,在各个任务领域都取得了惊人的成就。与视觉识别和语言模型不同,大规模通用用户表示的研究仍然有待探索。在这里,我们探讨了通用用户表示学习的可能性,在大规模训练的通用用户编码器。我们证明了缩放定律存在于用户表示学习区域中,其中训练误差与计算量成幂律关系。我们的对比学习用户编码器(CLUE)优化了与任务无关的目标,由此产生的用户嵌入扩展了我们对各种下游任务中可能做的事情的期望。CLUE还表现出很大的可移植性,以其他领域和公司,在一个在线实验的表现显示,点击率(CTR)显着改善。此外,我们还研究了模型性能如何受到比例因子的影响,如训练数据大小,模型容量,序列长度和批量大小。最后,我们讨论了一般的CLUE的更广泛的影响。

英文摘要:

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑