VOLD：通过在线蒸馏将LLM推理能力转移到视觉语言模型
VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation
AI总结：
本文提出VOLD框架，通过在线蒸馏将文本模型的推理能力转移到视觉语言模型，利用组相对策略优化与在线蒸馏结合，提升推理性能，并验证了冷启动对齐在在线训练中的重要性。
摘要：
训练视觉语言模型（VLMs）进行复杂推理仍是一项具有挑战性的任务，例如由于高质量图像-文本推理数据稀缺。相反，基于文本的推理资源丰富且可扩展，但如何利用它们来增强VLM推理仍是一个开放性问题。为此，我们提出了VOLD，一种将推理能力从文本-only教师模型转移到VLM学生模型的框架。为此，VOLD结合了通过组相对策略优化（GRPO）进行的强化学习与在线蒸馏，使学生推理轨迹能够由教师模型引导，从而在单独使用GRPO时获得显著提升。我们进一步表明，在此场景中，在线训练阶段有效的转移需要冷启动对齐，并且在教师和学生之间缺乏足够的分布对齐时，在线蒸馏无法提供有意义的指导。我们评估了VOLD在MMMU-Pro、MathVision、MathVista和LogicVista等多样化的基准测试中，显示出VOLD显著优于基线模型，并在现有最先进水平上取得显著提升。我们的消融研究显示了通过SFT进行冷启动对齐在文本-only教师与在线蒸馏中的重要性。
英文摘要：
Training vision-language models (VLMs) for complex reasoning remains a challenging task, i.a. due to the scarcity of high-quality image-text reasoning data. Conversely, text-based reasoning resources are abundant and scalable, but it is still an open question how to leveraging them for VLM reasoning. To address this problem, we propose VOLD, a framework to transfer reasoning capabilities from text-only teacher models to VLM student models. To this end, VOLD combines reinforcement learning via Group Relative Policy Optimization (GRPO) with on-policy distillation, which allows the student reasoning traces to be guided by the teacher model, resulting in a significant gain over using GRPO alone. We further show that a cold-start alignment is essential for an effective transfer during the online training phase in this scenario and that without sufficient distributional alignment between teacher and student, on-policy distillation fails to provide meaningful guidance. We evaluate VOLD across diverse benchmarks including MMMU-Pro, MathVision, MathVista, and LogicVista, showing that VOLD outperforms the baseline model significantly and improves over the state of the art by a margin. Our ablation shows the importance of a cold-start alignment via SFT for on-policy distillation with a text-only teacher.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
