学习 deliberation：基于多智能体强化学习的元策略协作用于代理语言模型
Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning
AI总结：
本文提出元策略推理框架MPDF，结合SoftRankPO算法，通过学习动态推理策略提升多智能体LLM在复杂推理任务中的性能。
摘要：
多智能体系统中的大型语言模型（LLMs）在复杂推理中展现出潜力，但其效果常受到固定协作协议的限制。这些框架通常侧重于宏观层面的协调，而忽视了智能体内部的推理能力。这种关键的元认知盲点将智能体视为被动执行者，无法根据内部认知状态如不确定性或信心调整其策略。我们引入了元策略推理框架（MPDF），其中智能体在一组高层元认知动作上学习一个去中心化的策略：坚持、细化和让步。为克服传统策略梯度在这一设置中的不稳定性，我们开发了SoftRankPO，一种新的强化学习算法。SoftRankPO通过基于奖励排名的平滑正常极量来塑造优势，使学习过程对奖励方差具有鲁棒性。实验显示，MPDF结合SoftRankPO在五个数学和一般推理基准测试中相比六种最先进的启发式和学习多智能体推理算法，平均准确率提高了4-5%。我们的工作提出了一种学习适应性、元认知策略的多智能体LLM系统范式，将焦点从设计固定协议转向学习动态、推理策略。
英文摘要：
Multi-agent systems of large language models (LLMs) show promise for complex reasoning, but their effectiveness is often limited by fixed collaboration protocols. These frameworks typically focus on macro-level orchestration while overlooking agents' internal deliberative capabilities. This critical meta-cognitive blindspot treats agents as passive executors unable to adapt their strategy based on internal cognitive states like uncertainty or confidence. We introduce the Meta-Policy Deliberation Framework (MPDF), where agents learn a decentralized policy over a set of high-level meta-cognitive actions: Persist, Refine, and Concede. To overcome the instability of traditional policy gradients in this setting, we develop SoftRankPO, a novel reinforcement learning algorithm. SoftRankPO stabilizes training by shaping advantages based on the rank of rewards mapped through smooth normal quantiles, making the learning process robust to reward variance. Experiments show that MPDF with SoftRankPO achieves a a 4-5% absolute gain in average accuracy across five mathematical and general reasoning benchmarks compared to six state-of-the-art heuristic and learning-based multi-agent reasoning algorithms. Our work presents a paradigm for learning adaptive, meta-cognitive policies for multi-agent LLM systems, shifting the focus from designing fixed protocols to learning dynamic, deliberative strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
