arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.14729cs.IRcs.AI
收藏

面向推荐的无偏知识蒸馏

Unbiased Knowledge Distillation for Recommendation

University of Science and Technology of China(中国科学技术大学);Zhejiang University(浙江大学)

Gang Chen, Jiawei Chen, Fuli Feng, Sheng Zhou, Xiangnan He

摘要:

知识蒸馏(KD)作为一种有前途的模型压缩方法,已被应用于推荐系统(RS)中以减少推理延迟。传统的解决方案首先从训练数据中训练一个完整的教师模型,然后转移其知识(\ie \textit{soft labels})来监督紧凑的学生模型的学习。然而,我们发现这样一个标准的蒸馏范式会产生严重的偏见问题-流行的项目更强烈的推荐后,蒸馏。这种效应会阻止学生模型做出准确和公平的推荐,从而降低RS的有效性。在这项工作中,我们确定了KD的偏见的起源-它根源于有偏见的软标签从老师,并进一步传播和加强在蒸馏。为了纠正这一点,我们提出了一个新的KD方法与分层蒸馏策略。它首先根据项目的受欢迎程度将其划分为多个组,然后提取每个组中的排名知识来监督学生的学习。我们的方法很简单,教师不可知-它在蒸馏阶段工作,而不影响教师模型的训练。我们进行了广泛的理论和实证研究,以验证我们的建议的有效性。我们在www.example.com上发布代码。

英文摘要:

As a promising solution for model compression, knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge (\ie \textit{soft labels}) to supervise the learning of a compact student model. However, we find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation. This effect prevents the student model from making accurate and fair recommendations, decreasing the effectiveness of RS. In this work, we identify the origin of the bias in KD -- it roots in the biased soft labels from the teacher, and is further propagated and intensified during the distillation. To rectify this, we propose a new KD method with a stratified distillation strategy. It first partitions items into multiple groups according to their popularity, and then extracts the ranking knowledge within each group to supervise the learning of the student. Our method is simple and teacher-agnostic -- it works on distillation stage without affecting the training of the teacher model. We conduct extensive theoretical and empirical studies to validate the effectiveness of our proposal. We release our code at: https://github.com/chengang95/UnKD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑