自问式视觉语言模型：用于组合视觉推理的强化学习
Self-Questioning Vision-Language Models: Reinforcement Learning for Compositional Visual Reasoning
AI总结：
提出自问式框架，通过GRPO强化学习训练VLM自动分解问题并回答子问题，提升组合视觉推理能力，在CLEVR和A-OKVQA上验证有效性。
摘要：
视觉语言模型（VLM）是处理图像和文本的AI系统，但它们通常难以处理需要多步骤链式推理的组合视觉推理问题，例如识别物体、计数并比较结果。现有方法通过训练模型使用人工编写的逐步解释来改进推理，但创建这些注释成本高昂且难以扩展。我们提出一个自问式框架，使用称为组相对策略优化（GRPO）的强化学习算法，训练VLM将视觉问题分解为更小的子问题，并在生成最终答案前回答每个子问题。模型从未见过如何分解问题的示例，而是通过奖励信号（根据输出是否包含子问题以及最终答案是否正确评分）自行发现这种行为。我们将该框架应用于一个30亿参数的模型，在合成几何形状场景（CLEVR）和真实世界照片（A-OKVQA）上进行训练。在A-OKVQA上，自问式和标准强化学习均显著提高了未训练模型的准确率（分别为52.2%和51.6%，对比46.8%）。我们引入了首个自问式VLM，不仅像标准RL那样奖励最终答案，还额外奖励生成中间子问题，使其能够发现组合分解策略。这些结果表明，教会AI系统自问中间问题是复杂视觉推理的一种有前景的策略，特别是当问题难度需要显式的逐步分解时。
英文摘要：
Vision-Language Models (VLMs) are AI systems that process both images and text, yet they often struggle with compositional visual reasoning questions that require chaining multiple steps together, such as identifying objects, counting them, and comparing the results. Existing approaches improve this reasoning by training models on human-written step-by-step explanations, but creating these annotations is expensive and difficult to scale. We propose a self-questioning framework that trains a VLM to break visual questions into smaller sub-questions and answer each one before producing a final response, using a reinforcement learning algorithm called Group Relative Policy Optimization (GRPO). The model is never shown examples of how to decompose questions, it discovers this behavior on its own, guided by a reward signal that scores whether the output contains sub-questions and whether the final answer is correct. We apply this framework to a 3-billion-parameter model, training on both synthetic scenes of geometric shapes (CLEVR) and real-world photographs (A-OKVQA). On A-OKVQA, both self-questioning and standard reinforcement learning substantially improve accuracy over the untrained model (52.2% and 51.6% vs. 46.8%). We introduce the first self-questioning VLM by rewarding not only the final answer like standard RL but additionally for generating intermediate sub-questions, enabling it to discover compositional decomposition strategies. These results suggest that teaching AI systems to ask themselves intermediate questions is a promising strategy for complex visual reasoning, particularly when the difficulty of a question warrants explicit step-by-step decomposition.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
