跨模态知识蒸馏与WordNet-松弛文本嵌入用于鲁棒图像分类
Crossmodal Knowledge Distillation with WordNet-Relaxed Text Embeddings for Robust Image Classification
AI总结：
提出一种多教师跨模态KD框架，通过结合CLIP图像嵌入与可学习的WordNet-松弛文本嵌入，缓解标签泄露并引入丰富文本线索，显著提升图像分类性能。
摘要：
跨模态知识蒸馏（KD）旨在利用多模态教师模型增强单模态学生模型。特别地，当教师的模态包含学生的模态时，可利用额外的互补信息来改善知识迁移。在有监督图像分类中，图像数据集通常包含代表高级概念的类别标签，这为融入文本线索进行跨模态KD提供了自然途径。然而，这些标签极少能捕捉真实世界视觉中的深层语义结构，若直接作为输入使用会导致标签泄露，最终限制KD性能。为解决这些问题，我们提出一种多教师跨模态KD框架，在分层损失下将CLIP图像嵌入与可学习的WordNet-松弛文本嵌入相结合。通过避免直接使用确切类名，转而使用语义更丰富的WordNet扩展，我们缓解了标签泄露并引入了更多样化的文本线索。实验表明，该策略显著提升了学生模型性能，而嘈杂或过于精确的文本嵌入会阻碍蒸馏效率。可解释性分析证实，WordNet-松弛提示促使模型更依赖视觉特征而非文本捷径，同时仍能有效融入新引入的文本线索。我们的方法在六个公开数据集上取得了最先进或第二佳的结果，证明了其在推进跨模态KD方面的有效性。
英文摘要：
Crossmodal knowledge distillation (KD) aims to enhance a unimodal student using a multimodal teacher model. In particular, when the teacher's modalities include the student's, additional complementary information can be exploited to improve knowledge transfer. In supervised image classification, image datasets typically include class labels that represent high-level concepts, suggesting a natural avenue to incorporate textual cues for crossmodal KD. However, these labels rarely capture the deeper semantic structures in real-world visuals and can lead to label leakage if used directly as inputs, ultimately limiting KD performance. To address these issues, we propose a multi-teacher crossmodal KD framework that integrates CLIP image embeddings with learnable WordNet-relaxed text embeddings under a hierarchical loss. By avoiding direct use of exact class names and instead using semantically richer WordNet expansions, we mitigate label leakage and introduce more diverse textual cues. Experiments show that this strategy significantly boosts student performance, whereas noisy or overly precise text embeddings hinder distillation efficiency. Interpretability analyses confirm that WordNet-relaxed prompts encourage heavier reliance on visual features over textual shortcuts, while still effectively incorporating the newly introduced textual cues. Our method achieves state-of-the-art or second-best results on six public datasets, demonstrating its effectiveness in advancing crossmodal KD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
