Agent探索性策略优化用于多模态Agent推理
Agent Explorative Policy Optimization for Multimodal Agentic Reasoning
AI总结：
针对多模态Agent推理中思考与工具使用的不对称性（Thinking-Acting Gap），提出AXPO算法，通过固定思考前缀并重采样工具调用及其延续，结合基于不确定性的前缀选择，显著提升工具使用率和模型性能。
摘要：
具有扩展推理能力的视觉-语言模型在复杂问题上取得成功，但许多现实问题需要外部工具，而仅靠内部推理往往无法解决。因此，Agent推理交织了两种具有结构不对称性的行为：思考（自包含的默认行为）和工具使用（高方差辅助行为）。我们将这种不对称性称为思考-行动差距。在标准RL方法（如GRPO）下，该差距在训练中表现为两种诊断症状：工具使用仅在约30%的rollout中被尝试，而当尝试时，组内使用工具的rollout在约40%的问题上全部错误，从而抑制了需要工具调用的学习信号。我们提出AXPO（Agent探索性策略优化）：对于每个全部错误的工具使用子组，AXPO固定思考前缀并重采样工具调用及其延续，同时结合基于不确定性的前缀选择。在九个多模态基准测试和三种规模的Qwen3-VL-Thinking上，SFT+AXPO在平均性能上优于SFT+GRPO（8B模型平均Pass@1提升1.8个百分点，Pass@4提升1.8个百分点），并且8B的SFT+AXPO在Pass@4上以4倍更少的参数超越了32B Base模型。
英文摘要：
Vision-language models with extended reasoning succeed on complex problems, but many real-world problems require external tools that internal reasoning alone often cannot resolve. Agentic reasoning therefore interleaves two behaviors with a structural asymmetry: thinking (the self-contained default) and tool use (a high-variance auxiliary acting). We refer to this asymmetry as the Thinking-Acting Gap. Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training: tool use is attempted on only ~30% of rollouts, and when attempted, the tool-using rollouts within a group are all-wrong on ~40% of questions, suppressing the learning signal at the tool calls that needed it. We propose AXPO (Agent eXplorative Policy Optimization): for each all-wrong tool-using subgroup, AXPO fixes the thinking prefix and resamples the tool call and its continuation, paired with uncertainty-based prefix selection. Across nine multimodal benchmarks and three scales of Qwen3-VL-Thinking, SFT+AXPO outperforms SFT+GRPO at average (+1.8pp Pass@1 and +1.8pp Pass@4 at 8B on average) and 8B with SFT+AXPO surpasses the 32B Base on Pass@4 with 4 times fewer parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
