当更好的教师不培养更好的学生：重新审视用于CLIP模型的KL知识蒸馏在视觉问答中的应用
When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
AI总结：
本文重新审视了CLIP模型在视觉问答中的知识蒸馏，发现更强教师不必然产生更好学生，揭示了现有蒸馏框架的局限性，并指明了参数高效多模态模型的设计新方向。
摘要：
视觉-语言模型（VLMs）在多模态任务中取得了显著成功，但其庞大的计算需求阻碍了高效部署。知识蒸馏（KD）已 emerged 作为一种构建轻量但竞争性模型的强大方法，在语言和视觉领域都有强有力的证据支持。然而，其在VLMs中的应用，特别是CLIP式模型，仍然有限，通常受限于小规模教师和狭窄的评估任务，如分类或检索。在本工作中，我们首次系统地研究了在一系列CLIP式教师模型上进行蒸馏，从标准基线到大规模的最先进模型。与在NLP和视觉领域观察到的趋势相反，我们发现更强的教师并不总是产生更好的学生；事实上，现有的蒸馏框架往往无法扩展，导致在视觉问答等下游多模态任务中的性能下降。我们的发现挑战了KD领域的主流假设，并指明了设计参数高效的多模态模型的新方向。
英文摘要：
Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to small-scale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
