Competitive Distillation：一种用于改进视觉分类的简单学习策略
Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
AI总结：
本文提出竞争蒸馏策略，通过让一组网络根据性能动态充当教师并参与竞争优化，同时引入随机扰动激励突变，有效提升了视觉分类任务的性能。
摘要：
深度神经网络（DNNs）显著推动了计算机视觉领域的发展。为了改进DNN训练过程，知识蒸馏方法通过引入从教师网络到学生网络的固定学习方向，展现了其在加速网络训练方面的有效性。在此背景下，提出了几种基于蒸馏的优化策略，例如深度互学习和自蒸馏，试图通过多个网络的协同训练实现通用的训练性能提升。然而，由于对网络间不同迭代中学习方向影响的理解不足，这些策略取得的改进有限。在本文中，我们提出了一种新颖的竞争蒸馏策略，允许一组中的每个网络根据其性能潜在地充当教师，从而增强整体学习性能。竞争蒸馏组织一组网络执行共享任务并参与竞争，其中提出了竞争优化来改进参数更新过程。我们进一步在竞争蒸馏中引入随机扰动，旨在激励网络产生突变以实现更好的视觉表示和全局最优。实验结果表明，竞争蒸馏在多样化任务和数据集上取得了有前景的性能。
英文摘要：
Deep Neural Networks (DNNs) have significantly advanced the field of computer vision. To improve DNN training process, knowledge distillation methods demonstrate their effectiveness in accelerating network training by introducing a fixed learning direction from the teacher network to student networks. In this context, several distillation-based optimization strategies are proposed, e.g., deep mutual learning and self-distillation, as an attempt to achieve generic training performance enhancement through the cooperative training of multiple networks. However, such strategies achieve limited improvements due to the poor understanding of the impact of learning directions among networks across different iterations. In this paper, we propose a novel competitive distillation strategy that allows each network in a group to potentially act as a teacher based on its performance, enhancing the overall learning performance. Competitive distillation organizes a group of networks to perform a shared task and engage in competition, where competitive optimization is proposed to improve the parameter updating process. We further introduce stochastic perturbation in competitive distillation, aiming to motivate networks to induce mutations to achieve better visual representations and global optimum. The experimental results show that competitive distillation achieves promising performance in diverse tasks and datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
