面向无约束视频的鲁棒跨模态知识蒸馏
Robust Cross-Modal Knowledge Distillation for Unconstrained Videos
AI总结：
针对无约束视频中模态噪声与语义关联不一致的问题，提出MNF净化教师模态并结合CSC进行自适应对比蒸馏，提升动作识别、视频检索及音频标注性能。
摘要：
跨模态蒸馏已被广泛用于在不同模态之间迁移知识，从而丰富目标单模态的表示。近期研究高度依赖视觉与声音之间的时间同步性，将其作为跨模态蒸馏的语义一致性。然而，在无约束视频中，由于不相关的模态噪声和差异化的语义关联，来自同步性的这种语义一致性很难得到保证。为此，我们首先提出一个 Modality Noise Filter（MNF，模态噪声滤波器）模块，利用跨模态上下文消除教师模态中的不相关噪声。在此净化之后，我们随后设计 Contrastive Semantic Calibration（CSC，对比语义校准）模块，通过以对比方式参考差异化的逐样本语义关联，为目标模态自适应地蒸馏有用知识。大量实验表明，在视觉动作识别和视频检索任务中，与其他蒸馏方法相比，我们的方法能够带来性能提升。我们还将其扩展到音频标注任务，以证明该方法的泛化能力。源代码可在 https://github.com/GeWu-Lab/cross-modal-distillation 获取。
英文摘要：
Cross-modal distillation has been widely used to transfer knowledge across different modalities, enriching the representation of the target unimodal one. Recent studies highly relate the temporal synchronization between vision and sound to the semantic consistency for cross-modal distillation. However, such semantic consistency from the synchronization is hard to guarantee in unconstrained videos, due to the irrelevant modality noise and differentiated semantic correlation. To this end, we first propose a \textit{Modality Noise Filter} (MNF) module to erase the irrelevant noise in teacher modality with cross-modal context. After this purification, we then design a \textit{Contrastive Semantic Calibration} (CSC) module to adaptively distill useful knowledge for target modality, by referring to the differentiated sample-wise semantic correlation in a contrastive fashion. Extensive experiments show that our method could bring a performance boost compared with other distillation methods in both visual action recognition and video retrieval task. We also extend to the audio tagging task to prove the generalization of our method. The source code is available at \href{https://github.com/GeWu-Lab/cross-modal-distillation}{https://github.com/GeWu-Lab/cross-modal-distillation}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
