Visual Program Distillation：将工具与程序化推理蒸馏进视觉-语言模型
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models
AI总结：
本文提出 Visual Program Distillation（VPD），通过采样、执行并验证 LLM 生成的视觉程序，将正确推理链蒸馏为 VLM 的单次前向推理能力，并在多个视觉问答与推理基准上取得最先进结果。
摘要：
解决诸如“右边的乐器是谁发明的？”这类复杂视觉任务，需要组合多项技能：理解空间、识别乐器，以及检索先验知识。近期工作通过使用大型语言模型（LLM）将此类任务分解为调用专门视觉模型的可执行程序，展现出良好前景。然而，生成的程序容易出错：它们会遗漏必要步骤、包含虚假步骤，并且在专门模型给出错误输出时无法恢复。此外，这些方法需要加载多个模型，导致较高的延迟和计算成本。我们提出 Visual Program Distillation（VPD），这是一个指令微调框架，可生成能够通过单次前向传播解决复杂视觉任务的视觉-语言模型（VLM）。VPD 蒸馏 LLM 的推理能力：先利用 LLM 采样多个候选程序，随后执行并验证这些程序，以找出正确程序。它将每个正确程序转换为对推理步骤的语言描述，再将这些描述蒸馏进一个 VLM。大量实验表明，VPD 提升了 VLM 的计数、空间关系理解和组合推理能力。我们经 VPD 训练的 PaLI-X 优于此前所有 VLM，在 MMBench、OK-VQA、A-OKVQA、TallyQA、POPE 和 Hateful Memes 等复杂视觉任务上取得最先进性能。由人工标注者参与的评估也证实，VPD 提高了模型响应的事实性和一致性。最后，内容审核方面的实验表明，VPD 还有助于在数据有限的情况下适配真实世界应用。
英文摘要：
Solving complex visual tasks such as "Who invented the musical instrument on the right?" involves a composition of skills: understanding space, recognizing instruments, and also retrieving prior knowledge. Recent work shows promise by decomposing such tasks using a large language model (LLM) into an executable program that invokes specialized vision models. However, generated programs are error-prone: they omit necessary steps, include spurious ones, and are unable to recover when the specialized models give incorrect outputs. Moreover, they require loading multiple models, incurring high latency and computation costs. We propose Visual Program Distillation (VPD), an instruction tuning framework that produces a vision-language model (VLM) capable of solving complex visual tasks with a single forward pass. VPD distills the reasoning ability of LLMs by using them to sample multiple candidate programs, which are then executed and verified to identify a correct one. It translates each correct program into a language description of the reasoning steps, which are then distilled into a VLM. Extensive experiments show that VPD improves the VLM's ability to count, understand spatial relations, and reason compositionally. Our VPD-trained PaLI-X outperforms all prior VLMs, achieving state-of-the-art performance across complex vision tasks, including MMBench, OK-VQA, A-OKVQA, TallyQA, POPE, and Hateful Memes. An evaluation with human annotators also confirms that VPD improves model response factuality and consistency. Finally, experiments on content moderation demonstrate that VPD is also helpful for adaptation to real-world applications with limited data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
