LinguDistill: 通过选择性跨模态蒸馏恢复视觉语言模型中的语言能力
LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation
AI总结：
本文提出LinguDistill方法，通过利用冻结的原始语言模型作为教师，选择性蒸馏语言信号以恢复语言能力，同时保持视觉基础，提升了语言和知识基准性能。
摘要：
将预训练语言模型（LMs）适应为视觉语言模型（VLMs）时，由于多模态适应过程中引入的表示偏移和跨模态干扰，会降低其固有语言能力。这种损失难以通过标准目标的任务特定微调恢复。先前的恢复方法通常引入额外模块作为中间对齐层以维持或隔离模态特定子空间，这会增加架构复杂性，增加推理时的参数，并限制模型和设置之间的灵活性。我们提出LinguDistill，一种无需适配器的蒸馏方法，通过利用冻结的原始LM作为教师来恢复语言能力。我们通过引入逐层KV缓存共享，克服了使教师能够提供视觉条件的教师监督的关键挑战，这使教师能够接触到学生多模态表示，而无需修改任一模型的架构。然后，我们选择性地蒸馏教师在语言密集数据上的强语言信号以恢复语言能力，同时保持学生在多模态任务上的视觉基础。结果，LinguDistill在语言和知识基准上恢复了约10%的性能损失，同时在视觉密集任务上保持了可比的性能。我们的发现表明，可以通过不增加额外模块的方式恢复语言能力，为多模态模型中的模态特定退化提供了一种高效且实用的解决方案。
英文摘要：
Turning a pretrained language model (LM) into a vision-language model (VLM) through multimodal fine-tuning often erodes its native language ability, a form of catastrophic forgetting that shows up even on text-only tasks. This loss is hard to undo with further fine-tuning, and existing remedies add adapters or alignment modules that increase architectural complexity and inference cost. We propose LinguDistill, an adapter-free knowledge distillation method that uses the original frozen LM as the teacher during multimodal post-training. To let a text-only teacher supervise vision-conditioned outputs, we introduce layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without changing either architecture. We then apply distillation selectively, on language-heavy data only, so the teacher restores linguistic ability while the student keeps its visual grounding on document and OCR tasks. LinguDistill recovers the language and knowledge performance lost during multimodal fine-tuning, matching the original VLM on average over text-only benchmarks (ARC, HellaSwag) and exceeding it on ScienceQA, while keeping vision-heavy performance close to standard fine-tuning. Since the teacher is dropped after training, the final model adds no parameters and no inference cost. More broadly, our results show that a model's own pre-adaptation backbone is a practical teacher for undoing forgetting, suggesting a simple recipe for keeping language ability intact as models are extended to new modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
