arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.01130cs.IRcs.AI
收藏

用于Top-K推荐的异构模型蒸馏

Distillation from Heterogeneous Models for Top-K Recommendation

Pohang University of Science and Technology (POSTECH)(浦项科技大学);Yonsei University(延世大学);Microsoft Research Asia(微软亚洲研究院)

SeongKu Kang, Wonbin Kweon, Dongha Lee, Jianxun Lian, Xing Xie, Hwanjo Yu

摘要:

最近的推荐系统已经显示出显着的性能,通过使用一个集成的异构模型。然而,它的成本非常高,因为它需要与模型数量成比例的资源和推理延迟,这仍然是生产的瓶颈。我们的工作旨在通过知识蒸馏(KD)将异构教师的集成知识转换为轻量级的学生模型,以在保持高精度的同时减少巨大的推理成本。通过实证研究发现,异质性教师的知识迁移会严重降低提炼效能。然而,我们表明,一个重要的信号,以减轻困难,可以从教师的培训轨迹。本文提出了一个新的知识分配框架,命名为HetComp,通过转移从教师的轨迹生成的知识的易到难的序列来指导学生模型。为了根据学生的学习状态提供指导,HetComp使用动态知识构建来提供逐步困难的排名知识和自适应知识转移来逐步转移更细粒度的排名信息。我们的综合实验表明,HetComp显着提高蒸馏质量和学生模型的泛化。

英文摘要:

Recent recommender systems have shown remarkable performance by using an ensemble of heterogeneous models. However, it is exceedingly costly because it requires resources and inference latency proportional to the number of models, which remains the bottleneck for production. Our work aims to transfer the ensemble knowledge of heterogeneous teachers to a lightweight student model using knowledge distillation (KD), to reduce the huge inference costs while retaining high accuracy. Through an empirical study, we find that the efficacy of distillation severely drops when transferring knowledge from heterogeneous teachers. Nevertheless, we show that an important signal to ease the difficulty can be obtained from the teacher's training trajectory. This paper proposes a new KD framework, named HetComp, that guides the student model by transferring easy-to-hard sequences of knowledge generated from the teachers' trajectories. To provide guidance according to the student's learning state, HetComp uses dynamic knowledge construction to provide progressively difficult ranking knowledge and adaptive knowledge transfer to gradually transfer finer-grained ranking information. Our comprehensive experiments show that HetComp significantly improves the distillation quality and the generalization of the student model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑