arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.17890cs.IRcs.CLcs.LG
收藏

SLMRec:将大语言模型蒸馏为小模型用于序列推荐

SLMRec: Distilling Large Language Models into Small for Sequential Recommendation

Rutgers University(新泽西罗格斯大学);Broad Institute of MIT and Harvard(哈佛大学与麻省理工学院Broad研究所);Ant Group(蚂蚁集团);Dian Diagnostics Group Co.(迪安诊断集团);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Technology Sydney(悉尼技术大学)

Wujiang Xu, Qitian Wu, Zujie Liang, Jiaojiao Han, Xuying Ning, Yunxiao Shi, Wenfang Lin, Yongfeng Zhang

AI总结:

本文提出SLMRec,通过知识蒸馏将大语言模型压缩为小模型用于序列推荐,仅用13%参数即达最佳性能,训练和推理分别提速6.6倍和8.0倍,并给出理论解释。

摘要:

序列推荐(SR)任务涉及根据用户过去的行为预测其下一个可能交互的项目。SR模型通过检查用户的行为序列来识别更复杂的行为模式和时间动态。近期研究表明,大语言模型(LLM)对序列推荐系统产生了巨大影响,要么将序列推荐视为语言建模,要么作为用户表示的主干网络。尽管这些方法取得了出色的性能,但几乎没有证据表明大语言模型的必要性以及需要多大的语言模型,尤其是在序列推荐场景中。同时,由于LLM的巨大规模,在通常需要每天处理数十亿流量日志的现实平台中,应用基于LLM的模型效率低下且不切实际。在本文中,我们通过在大型工业数据集上进行广泛实验,探索了LLM深度的影响。令人惊讶的是,我们的动机实验表明,LLM的大部分中间层是冗余的,表明剪除其余层仍能保持强大的性能。受此洞察的启发,我们为SR赋能小语言模型,即SLMRec,它采用了一种简单而有效的知识蒸馏方法。此外,SLMRec与其他训练后效率技术(如量化和剪枝)正交,因此可以组合使用它们。综合实验结果表明,所提出的SLMRec模型仅使用基于LLM的推荐模型参数的13%即可获得最佳性能,同时在训练和推理时间成本上分别实现了高达6.6倍和8.0倍的加速。此外,我们为小语言模型在SR中为何能表现得与大语言模型相当提供了理论依据。

英文摘要:

Sequential Recommendation (SR) task involves predicting the next item a user is likely to interact with, given their past interactions. The SR models examine the sequence of a user's actions to discern more complex behavioral patterns and temporal dynamics. Recent research demonstrates the great impact of LLMs on sequential recommendation systems, either viewing sequential recommendation as language modeling or serving as the backbone for user representation. Although these methods deliver outstanding performance, there is scant evidence of the necessity of a large language model and how large the language model is needed, especially in the sequential recommendation scene. Meanwhile, due to the huge size of LLMs, it is inefficient and impractical to apply a LLM-based model in real-world platforms that often need to process billions of traffic logs daily. In this paper, we explore the influence of LLMs' depth by conducting extensive experiments on large-scale industry datasets. Surprisingly, our motivational experiments reveal that most intermediate layers of LLMs are redundant, indicating that pruning the remaining layers can still maintain strong performance. Motivated by this insight, we empower small language models for SR, namely SLMRec, which adopt a simple yet effective knowledge distillation method. Moreover, SLMRec is orthogonal to other post-training efficiency techniques, such as quantization and pruning, so that they can be leveraged in combination. Comprehensive experimental results illustrate that the proposed SLMRec model attains the best performance using only 13% of the parameters found in LLM-based recommendation models while simultaneously achieving up to 6.6x and 8.0x speedups in training and inference time costs, respectively. Besides, we provide a theoretical justification for why small language models can perform comparably to large language models in SR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑