知道答案还不够:修复大型视觉-语言模型中的推理路径失败
Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs
AI总结:
PSO通过优化推理路径选择,提升大型视觉-语言模型的推理准确性和稳定性。
摘要:
我们揭示了大型视觉-语言模型(LVLMs)中一个关键但长期未被深入探讨的缺陷:即使这些模型知道正确的答案,它们也经常通过错误的推理路径到达那里。核心问题不是知识的缺乏,而是在庞大的推理搜索空间中存在路径选择偏差。尽管LVLMs通常能够采样出正确的解决方案轨迹,但它们倾向于偏爱不稳定或逻辑不一致的路径,导致结果变得不稳定和不可靠。在众多模型中,Pass@K(大K)与Pass@1之间显著的差异提供了有力证据,表明此类失败主要源于推理错误而非无知。为了系统地研究和解决这一问题,我们提出了PSO(路径选择优化),一种两阶段的后训练框架,旨在提高现有LVLMs的推理性能和稳定性。第一阶段中,我们使用基于模板和答案的奖励进行组相对策略优化(GRPO),以培养结构化、逐步推理的过程。第二阶段中,我们进行在线偏好优化,其中模型从GRPO生成的数据中采样推理路径,自我评估这些路径,并调整自身以朝向被偏好的轨迹。错误或次优的路径同时存储在负反馈记忆(NRM)中作为硬负样本,这些样本会定期重新审视,以防止模型重复先前的错误,并促进持续的推理优化。广泛的实验表明,PSO有效剪枝无效的推理路径,显著提高推理准确性(平均提升7.4%),并产生更加稳定和一致的思维链。我们的代码将在https://github.com/aiming-lab/PSO上提供。
英文摘要:
We reveal a critical yet underexplored flaw in Large Vision-Language Models (LVLMs): even when these models know the correct answer, they frequently arrive there through incorrect reasoning paths. The core issue is not a lack of knowledge, but a path selection bias within the vast reasoning search space. Although LVLMs are often capable of sampling correct solution trajectories, they disproportionately favor unstable or logically inconsistent ones, leading to erratic and unreliable outcomes. The substantial disparity between Pass@K (with large K) and Pass@1 across numerous models provides compelling evidence that such failures primarily stem from misreasoning rather than ignorance. To systematically investigate and address this issue, we propose PSO (Path-Select Optimization), a two-stage post-training framework designed to enhance both the reasoning performance and stability of existing LVLMs. In the first stage, we employ Group Relative Policy Optimization (GRPO) with template and answer-based rewards to cultivate structured, step-by-step reasoning. In the second stage, we conduct online preference optimization, where the model samples reasoning paths from GRPO-generated data, self-evaluates them, and aligns itself toward the preferred trajectories. Incorrect or suboptimal paths are concurrently stored in a Negative Replay Memory (NRM) as hard negatives, which are periodically revisited to prevent the model from repeating prior mistakes and to facilitate continual reasoning refinement. Extensive experiments show that PSO effectively prunes invalid reasoning paths, substantially enhances reasoning accuracy (with 7.4% improvements on average), and yields more stable and consistent chains of thought. Our code will be available at https://github.com/aiming-lab/PSO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。