知识蒸馏综合综述
A Comprehensive Survey on Knowledge Distillation
AI总结：
本文全面综述了知识蒸馏（KD）方法，从蒸馏源、方案、算法、模态及应用等新视角分类探讨了扩散模型、基础模型、transformers和LLMs等最新子类别，并讨论了现有挑战与未来方向。
摘要：
深度神经网络（DNNs）在计算机视觉和自然语言处理领域取得了显著性能，在学术界和工业界有各种应用。然而，随着DNNs和具有大量参数的transformer模型的最新进展，将这些大模型部署在边缘设备上会导致高运行时间和内存消耗等严重问题。这对于近期的大规模基础模型、视觉语言模型（VLMs）和大语言模型（LLMs）尤为令人担忧。知识蒸馏（KD）是使用师生架构解决上述问题的一种突出技术。更具体地说，使用来自笨重教师模型的额外知识训练轻量级学生模型。在这项工作中，提出了对知识蒸馏方法的全面综述。这包括从不同方面回顾KD：蒸馏源、蒸馏方案、蒸馏算法、按模态蒸馏、蒸馏应用以及现有方法之间的比较。与大多数现有综述（要么过时，要么仅仅更新以前的综述）不同，这项工作提出了一项具有新观点和表示结构的全面综述，对知识蒸馏的最新方法进行了分类和调查。本综述考虑了各种至关重要的子类别，包括用于扩散模型、3D输入、基础模型、transformers和LLMs的KD。此外，讨论了KD中现有的挑战和可能的未来研究方向。项目Github页面：https://github.com/IPL-Sharif/KD_Survey
英文摘要：
Deep Neural Networks (DNNs) have achieved notable performance in the fields of computer vision and natural language processing with various applications in both academia and industry. However, with recent advancements in DNNs and transformer models with a tremendous number of parameters, deploying these large models on edge devices causes serious issues such as high runtime and memory consumption. This is especially concerning with the recent large-scale foundation models, Vision-Language Models (VLMs), and Large Language Models (LLMs). Knowledge Distillation (KD) is one of the prominent techniques proposed to address the aforementioned problems using a teacher-student architecture. More specifically, a lightweight student model is trained using additional knowledge from a cumbersome teacher model. In this work, a comprehensive survey of knowledge distillation methods is proposed. This includes reviewing KD from different aspects: distillation sources, distillation schemes, distillation algorithms, distillation by modalities, applications of distillation, and comparison among existing methods. In contrast to most existing surveys, which are either outdated or simply update former surveys, this work proposes a comprehensive survey with a new point of view and representation structure that categorizes and investigates the most recent methods in knowledge distillation. This survey considers various critically important subcategories, including KD for diffusion models, 3D inputs, foundational models, transformers, and LLMs. Furthermore, existing challenges in KD and possible future research directions are discussed. Github page of the project: https://github.com/IPL-Sharif/KD_Survey
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
