面向边缘设备上CNN与Vision Transformer知识蒸馏的最优权衡
Towards Optimal Trade-offs in Knowledge Distillation for CNNs and Vision Transformers at the Edge
AI总结：
本文系统研究了边缘设备上CNN与ViT知识蒸馏的四个关键方面，包括架构配置、学生模型大小、输入分辨率及下游微调，为AI从业者提供了优化KD过程的有效策略。
摘要：
本文探讨了卷积神经网络（CNN）和视觉Transformer（ViT）架构在知识蒸馏（KD）过程中的四个方面，特别是在计算能力受限的边缘设备上执行时。首先，我们对CNN和ViT架构之间的KD过程进行了比较分析，旨在阐明采用不同架构配置作为教师和学生模型的可行性和有效性，同时评估其性能和效率。其次，我们在保持KD时长不变的情况下，探讨了改变学生模型大小对准确性和推理速度的影响。第三，我们研究了使用更高分辨率图像对准确性、内存占用和计算工作量的影响。最后，我们考察了在KD之后对学生模型进行微调以适应特定下游任务所获得的性能改进。通过实证评估和分析，本研究为AI从业者提供了关于在边缘设备上最大化KD过程有效性的最优策略的见解。
英文摘要：
This paper discusses four facets of the Knowledge Distillation (KD) process for Convolutional Neural Networks (CNNs) and Vision Transformer (ViT) architectures, particularly when executed on edge devices with constrained processing capabilities. First, we conduct a comparative analysis of the KD process between CNNs and ViT architectures, aiming to elucidate the feasibility and efficacy of employing different architectural configurations for the teacher and student, while assessing their performance and efficiency. Second, we explore the impact of varying the size of the student model on accuracy and inference speed, while maintaining a constant KD duration. Third, we examine the effects of employing higher resolution images on the accuracy, memory footprint and computational workload. Last, we examine the performance improvements obtained by fine-tuning the student model after KD to specific downstream tasks. Through empirical evaluations and analyses, this research provides AI practitioners with insights into optimal strategies for maximizing the effectiveness of the KD process on edge devices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
