MST-Distill：用于跨模态知识蒸馏的专家混合模型
MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation
AI总结：
提出MST-Distill框架，通过专家混合和实例级路由解决跨模态知识蒸馏中的路径选择和知识漂移问题，在五个多模态数据集上超越现有方法。
摘要：
知识蒸馏作为一种高效的知识迁移技术，已在单模态场景中取得了显著成功。然而，在跨模态设置下，由于数据和统计异质性，传统蒸馏方法面临重大挑战，无法充分利用跨模态教师模型中嵌入的互补先验知识。本文通过实验揭示了现有方法中的两个关键问题：蒸馏路径选择和知识漂移。为解决这些局限性，我们提出了MST-Distill，一种新颖的跨模态知识蒸馏框架，其特点是采用专家混合模型。我们的方法在跨模态和多模态配置中集成了多样化的教师模型集合，并结合实例级路由网络，实现自适应和动态蒸馏。该架构有效超越了依赖单调静态教师模型的传统方法的限制。此外，我们引入了一个即插即用的掩蔽模块，该模块独立训练以抑制模态特定差异并重建教师表示，从而减轻知识漂移并增强迁移效果。在涵盖视觉、音频和文本的五个多样化多模态数据集上的大量实验表明，我们的方法在跨模态蒸馏任务中显著优于现有的最先进知识蒸馏方法。源代码可在 https://github.com/Gray-OREO/MST-Distill 获取。
英文摘要：
Knowledge distillation as an efficient knowledge transfer technique, has achieved remarkable success in unimodal scenarios. However, in cross-modal settings, conventional distillation methods encounter significant challenges due to data and statistical heterogeneities, failing to leverage the complementary prior knowledge embedded in cross-modal teacher models. This paper empirically reveals two critical issues in existing approaches: distillation path selection and knowledge drift. To address these limitations, we propose MST-Distill, a novel cross-modal knowledge distillation framework featuring a mixture of specialized teachers. Our approach employs a diverse ensemble of teacher models across both cross-modal and multimodal configurations, integrated with an instance-level routing network that facilitates adaptive and dynamic distillation. This architecture effectively transcends the constraints of traditional methods that rely on monotonous and static teacher models. Additionally, we introduce a plug-in masking module, independently trained to suppress modality-specific discrepancies and reconstruct teacher representations, thereby mitigating knowledge drift and enhancing transfer effectiveness. Extensive experiments across five diverse multimodal datasets, spanning visual, audio, and text, demonstrate that our method significantly outperforms existing state-of-the-art knowledge distillation methods in cross-modal distillation tasks. The source code is available at https://github.com/Gray-OREO/MST-Distill.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
