arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.13993cs.AIcs.CLcs.CV
收藏

用于视觉语言模型中物理推理的奖励设计

Reward Design for Physical Reasoning in Vision-Language Models

Pacific Northwest National Laboratory(太平洋西北国家实验室)

Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana

AI总结:

本文探讨了奖励设计对视觉语言模型物理推理的影响,通过对比不同奖励信号发现,基于准确性的奖励在多数领域表现最佳,而基于注意力权重的奖励提升了空间推理能力。

摘要:

对视觉输入的物理推理需要紧密整合视觉感知、领域知识和多步符号推理。然而,最先进的视觉语言模型(VLMs)在物理基准测试中仍远低于人类水平。尽管后训练算法如监督微调(SFT)和群体相对策略优化(GRPO)在语言模型中表现出强大的推理增益,但奖励设计如何塑造VLM的物理推理行为仍不明确。本文提出了一种系统性的奖励消融研究,用于GRPO基于的VLM训练中的物理推理。我们比较了四种递增语义丰富的奖励信号:格式合规性、答案准确性、一个综合评分奖励(答案正确性、物理原理识别和单位一致性)以及一种新的内部奖励,该奖励来源于模型对输入图像区域的注意力权重。我们使用IBM Granite Vision 3.3(2B)在PhyX基准测试上进行评估,该基准测试包含3000个问题,涵盖六个物理领域和六个推理类型,跨越多项选择和开放性格式。在两种格式中,GRPO与基于准确性的奖励在大多数领域优于SFT,尽管收益因奖励类型和领域而异。奖励设计并不统一提高性能。相反,它诱导了领域特定的推理行为。基于准确性的奖励提供了最强的整体增益。评分奖励提高了结构推理质量,但没有一致的准确性提升。基于注意力的奖励增强了空间推理能力,但损害了符号领域的表现。我们的内部注意力权重奖励不需要空间注释,并将空间关系准确性从0.27提升到0.50,表明在生成过程中监督模型的注意力位置是视觉基础物理推理的一个有前途的方向。

英文摘要:

Physical reasoning over images requires integrating perception, domain knowledge, and multi-step symbolic inference, yet Vision-Language Models (VLMs) fall far short of human performance on physics benchmarks. Reinforcement learning can improve reasoning via reward signals, but it remains unclear whether these signals must come from outside the model or can be derived from the model itself. We compare external rewards, which rely on an outside verifier (rule-based format and answer-correctness checks, or an LLM-judge rubric), with an internal attention-based reward computed from the model's own attention over image regions, requiring no labels or judge. We find that external rewards drive most of the accuracy gains, while preserving more coherent reasoning when rewarded for reasoning tokens. The internal attention-based reward, provides an annotation-free grounding signal that improves spatial-relation reasoning and is complementary to external rewards: adding it to format and accuracy rewards, at no additional annotation cost, improves average accuracy on Molmo~2 while generalizing to out-of-distribution tasks. Our results suggest that internal attention rewards can supplement external supervision and lead to improved physical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑