arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.08401cs.CLcs.AI
收藏

AIPO: 通过主动交互学习推理

AIPO: Learning to Reason from Active Interaction

Department of Data Science and AI, Faculty of Information Technology, Monash University, Australia(数据科学与人工智能系,信息科技学院,墨尔本大学,澳大利亚)

Junnan Liu, Linhao Luo, Thuy-Trang Vu, Gholamreza Haffari

AI总结:

AIPO通过主动多智能体交互提升大语言模型推理能力,引入三个协作代理解决推理瓶颈,改进探索效率并扩展能力边界。

摘要:

近期大语言模型(LLM)的进展展示了卓越的推理能力,主要受可验证奖励强化学习(RLVR)推动。然而,现有RL算法面临探索受限于策略模型固有能力边界的根本限制。尽管近期方法引入外部专家演示扩展此边界,但通常依赖完整轨迹级指导,样本效率低、信息稀疏且可能限制探索于静态指导空间。受多智能体系统的启发,我们提出AIPO,一种增强的强化学习框架,通过探索期间的主动多智能体交互提升LLM推理能力。具体而言,AIPO使策略模型在遇到推理瓶颈时主动咨询三个功能协作代理,即验证代理、知识代理和推理代理,从而获得细粒度和针对性的指导,主动扩展其能力边界。我们进一步引入定制的重要性采样系数和剪裁策略,以缓解从代理提供的反馈中学习时出现的离策略偏差和梯度消失问题。训练后,策略模型可独立进行推理而不依赖协作代理。在多样化的推理基准测试中,包括AIME、MATH500、GPQA-Diamond和LiveCodeBench,AIPO一致提升了推理性能,跨不同策略模型和RLVR算法具有鲁棒泛化能力,并有效扩展了策略模型的推理能力边界。

英文摘要:

Recent advances in LLMs have demonstrated strong reasoning capabilities, largely stimulated by RLVR. However, the exploration of existing RLVR algorithms remains largely constrained by the knowledge and reasoning strategies already accessible to the policy model. Although recent methods introduce external expert demonstrations to broaden exploration, they typically rely on complete trajectory-level guidance, which can be sample-inefficient, information-sparse, and insufficiently adaptive to intermediate reasoning bottlenecks. Inspired by collaborative multi-agent systems, we propose AIPO, an enhanced reinforcement learning framework that improves LLM reasoning through active multi-agent interaction during exploration. Specifically, when encountering reasoning bottlenecks, AIPO enables the policy model to proactively consult three functional collaborative agents, namely the verify agent, knowledge agent, and reasoning agent, thereby obtaining fine-grained and state-dependent guidance during rollout. The resulting mixed-policy trajectories expose the policy to reasoning directions that may be difficult to discover through isolated on-policy exploration. To learn effectively from collaborator-provided tokens, we further introduce a corrected importance sampling coefficient together with a lower-bound clipping strategy to mitigate off-policy discrepancy and vanishing gradients. After training, the policy model reasons independently without relying on collaborative agents. Extensive experiments across mathematical, scientific, coding, and puzzle reasoning benchmarks show that AIPO consistently improves reasoning performance and generalizes across different policy models, collaborator backbones, and RLVR algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑