arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.02870cs.IR
收藏

Top-K推荐系统的双向蒸馏

Bidirectional Distillation for Top-K Recommender System

Wonbin Kweon, SeongKu Kang, Hwanjo Yu

摘要:

推荐系统(RS)已经开始采用知识蒸馏,这是一种模型压缩技术,利用从繁琐的模型(教师)传递的知识训练紧凑的模型(学生)。最先进的方法依赖于单向蒸馏,只将知识从教师转移到学生,基本假设是教师总是优于学生。然而,我们证明,学生的表现比老师在一个显着的比例的测试集,特别是RS。基于这一观察,我们提出了双向蒸馏(BD)框架,教师和学生都协同提高彼此。具体而言,每个模型都使用蒸馏损失进行训练,使其遵循其他模型的预测以及其原始损失函数。为了有效的双向蒸馏,我们提出了秩差异感知抽样方案,只提取信息的知识,可以充分增强对方。该方案的目的是有效地应付教师和学生之间的巨大的性能差距。通过双向培训,与单独培训相比,教师和学生都得到了显着改善。我们在真实世界数据集上的广泛实验表明,我们提出的框架始终优于最先进的竞争对手。我们还提供分析,以深入了解BD和消融研究,以验证每个拟议组件的有效性。

英文摘要:

Recommender systems (RS) have started to employ knowledge distillation, which is a model compression technique training a compact model (student) with the knowledge transferred from a cumbersome model (teacher). The state-of-the-art methods rely on unidirectional distillation transferring the knowledge only from the teacher to the student, with an underlying assumption that the teacher is always superior to the student. However, we demonstrate that the student performs better than the teacher on a significant proportion of the test set, especially for RS. Based on this observation, we propose Bidirectional Distillation (BD) framework whereby both the teacher and the student collaboratively improve with each other. Specifically, each model is trained with the distillation loss that makes to follow the other's prediction along with its original loss function. For effective bidirectional distillation, we propose rank discrepancy-aware sampling scheme to distill only the informative knowledge that can fully enhance each other. The proposed scheme is designed to effectively cope with a large performance gap between the teacher and the student. Trained in the bidirectional way, it turns out that both the teacher and the student are significantly improved compared to when being trained separately. Our extensive experiments on real-world datasets show that our proposed framework consistently outperforms the state-of-the-art competitors. We also provide analyses for an in-depth understanding of BD and ablation studies to verify the effectiveness of each proposed component.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑