GTR：引导式思维强化防止基于RL的VLM智能体训练中的思维崩溃
GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Training
AI总结：
针对RLVR训练VLM智能体时仅依赖动作结果奖励导致的思维崩溃问题，提出GTR框架，通过自动校正器在每步强化学习中评估并优化推理，显著提升了模型性能与泛化能力且无需密集人工标注。
摘要：
带有可验证结果奖励的强化学习（RLVR）已有效扩展了大型语言模型（LLMs）中的思维链（CoT）推理。然而，其在视觉环境中训练视觉语言模型（VLM）智能体进行目标导向动作推理的有效性尚不明确。本研究通过在复杂纸牌游戏（如24点）和来自ALFWorld的具身任务上进行大量实验，探讨了这一问题。我们发现，当奖励仅基于动作结果时，RL无法激励VLMs中的CoT推理，反而会导致一种我们称之为思维崩溃的现象，其特征是智能体思维多样性的快速丧失、与状态无关且不完整的推理，以及随后的无效动作，从而导致负奖励。为了对抗思维崩溃，我们强调了过程指导的必要性，并提出了一种自动校正器，在RL的每一步评估并优化智能体的推理。这个简单且可扩展的GTR（引导式思维强化）框架同时训练推理和动作，无需密集的、每步的人工标注。我们的实验表明，GTR显著提升了LLaVA-7b模型在各种视觉环境中的性能和泛化能力，与SoTA模型相比，在模型规模明显更小的情况下，任务成功率提高了3-5倍。
英文摘要：
Reinforcement learning with verifiable outcome rewards (RLVR) has effectively scaled up chain-of-thought (CoT) reasoning in large language models (LLMs). Yet, its efficacy in training vision-language model (VLM) agents for goal-directed action reasoning in visual environments is less established. This work investigates this problem through extensive experiments on complex card games, such as 24 points, and embodied tasks from ALFWorld. We find that when rewards are based solely on action outcomes, RL fails to incentivize CoT reasoning in VLMs, instead leading to a phenomenon we termed thought collapse, characterized by a rapid loss of diversity in the agent's thoughts, state-irrelevant and incomplete reasoning, and subsequent invalid actions, resulting in negative rewards. To counteract thought collapse, we highlight the necessity of process guidance and propose an automated corrector that evaluates and refines the agent's reasoning at each RL step. This simple and scalable GTR (Guided Thought Reinforcement) framework trains reasoning and action simultaneously without the need for dense, per-step human labeling. Our experiments demonstrate that GTR significantly enhances the performance and generalization of the LLaVA-7b model across various visual environments, achieving 3-5 times higher task success rates compared to SoTA models with notably smaller model sizes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
