arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.09667cs.AIcs.CL
收藏

协同多智能体测试时间强化学习用于推理

Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoning

MIT(麻省理工学院);NUS(新加坡国立大学);NYU(纽约大学);Microsoft(微软);Columbia(哥伦比亚大学);NTU(国立科技大学)

Zhiyuan Hu, Yunhai Hu, Juncheng Liu, Shuyue Stella Li, Yucheng Wang, Zhen Xu, See-Kiong Ng, Anh Tuan Luu, Xinxing Xu, Bryan Hooi, Cynthia Breazeal, Hae Won Park

AI总结:

MATTRL通过在推理阶段注入结构化文本经验,提升多智能体在医学、数学和教育等领域的推理准确性,比多智能体基线提升3.67%,比单智能体基线提升8.67%。

摘要:

多智能体系统已发展为许多应用中的实用LLM驱动协作者,通过多样性和交叉验证获得鲁棒性。然而,多智能体强化学习(MARL)的训练是资源密集且不稳定的:协同适应队友会引入非平稳性,而奖励通常稀疏且高方差。因此,我们引入了多智能体测试时间强化学习(MATTRL),一种在推理时间将结构化文本经验注入多智能体协商的框架。MATTRL形成一个多专家团队,专门处理多轮讨论,检索并整合测试时间经验,并通过共识达成最终决策。我们还研究了信用分配,以构建一个回合级经验池,然后将其重新注入对话。在医学、数学和教育等具有挑战性的基准上,MATTRL在多智能体基线上的平均准确率提高了3.67%,在可比单智能体基线上的平均准确率提高了8.67%。消融研究检验了不同的信用分配方案,并详细比较了它们对训练结果的影响。MATTRL提供了一条无需调优的稳定、有效且高效的路径,以实现分布偏移鲁棒的多智能体推理。

英文摘要:

Multi-agent systems have evolved into practical LLM-driven collaborators for many applications, gaining robustness from diversity and cross-checking. However, multi-agent RL (MARL) training is resource-intensive and unstable: co-adapting teammates induce non-stationarity, and rewards are often sparse and high-variance. Therefore, we introduce \textbf{Multi-Agent Test-Time Reinforcement Learning (MATTRL)}, a framework that injects structured textual experience into multi-agent deliberation at inference time. MATTRL forms a multi-expert team of specialists for multi-turn discussions, retrieves and integrates test-time experiences, and reaches consensus for final decision-making. We also study credit assignment for constructing a turn-level experience pool, then reinjecting it into the dialogue. Across challenging benchmarks in medicine, math, and education, MATTRL improves accuracy by an average of 3.67\% over a multi-agent baseline, and by 8.67\% over comparable single-agent baselines. Ablation studies examine different credit-assignment schemes and provide a detailed comparison of how they affect training outcomes. MATTRL offers a stable, effective and efficient path to distribution-shift-robust multi-agent reasoning without tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑