Large Language Models (LLMs) have shown remarkable capabilities in general natural language processing tasks but often fall short in complex reasoning tasks. Recent studies have explored human-like problem-solving strategies, such as self-correct, to push further the boundary of single-model reasoning ability. In this work, we let a single model "step outside the box" by engaging multiple models to correct each other. We introduce a multi-agent collaboration strategy that emulates the academic peer review process. Each agent independently constructs its own solution, provides reviews on the solutions of others, and assigns confidence levels to its reviews. Upon receiving peer reviews, agents revise their initial solutions. Extensive experiments on three different types of reasoning tasks show that our collaboration approach delivers superior accuracy across all ten datasets compared to existing methods. Further study underscores the effectiveness of integrating confidence in reviews, demonstrates the superiority of feedback exchange over mere solution sharing, and highlights the role of capability and diversity in fostering successful collaboration.
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
UKP Lab, ATHENE National Research Center for Applied Cybersecurity, Technical University of Darmstadt(UKP实验室,ATHENE国家应用网络安全研究中心,德累斯顿技术大学);AIML Lab, Hessian Center for AI, Technical University of Darmstadt(AIML实验室,黑森人工智能中心,德累斯顿技术大学);Synthetic RNA Biology, Technical University of Darmstadt(合成RNA生物学,德累斯顿技术大学)
Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
Beihang University, Beijing, China(北京航空航天大学);Zhongguancun Laboratory, Beijing, China(中关村实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Hong Kong University of Science and Technology(香港科学与技术大学);Xi'an Jiaotong University, Xi'an, China(西安交通大学);Tsinghua University, Beijing, China(清华大学);Singapore University of Technology and Design(新加坡科技设计大学)
LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions
Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);Lambda Labs(Lambda实验室)
Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systems
National University of Singapore(国立新加坡大学);Singapore University of Technology(新加坡科技学院);Nanyang Technological University(南洋理工大学);Duke University(杜克大学);University of Waterloo(滑铁卢大学);Microsoft(微软公司);Peking University(北京大学);University of Pennsylvania(宾夕法尼亚大学)
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
Exploring Communication Strategies for Collaborative LLM Agents in Mathematical Problem-Solving
Institute for Intelligent Systems, University of Memphis, Memphis, TN, USA(智能系统研究所,密苏里大学);Department of Electrical and Computer Engineering, University of Memphis, Memphis, TN, USA(电气与计算机工程系,密苏里大学);AI4STEM Education Center, University of Georgia, Athens, GA, USA(AI4STEM教育中心,佐治亚大学);National GENIUS Center, Athens, GA, USA(国家GENIUS中心);Department of Mathematics, Science, and Social Studies Education, University of Georgia, Athens, GA, USA(数学、科学与社会科学教育系,佐治亚大学);Faculty of Education, The University of Hong Kong, Hong Kong, PR China(教育学院,香港大学);Educational Testing Service, Princeton, NJ, USA(教育测评服务);Department of Applied Social Sciences, Hong Kong Polytechnic University, Hong Kong, PR China(应用社会科学系,香港理工大学);Department of Psychology, University of Memphis, Memphis, TN, USA(心理学系,密苏里大学)
该研究使用 GPT-4o 在 MATH 数据集上比较双智能体数学解题的四种沟通模式,发现同伴协作准确率最高,并表明有效对话行为是提升 AI 教育协作解题的关键。
Judgment-Grounded Expansion for Peer Review Generation
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE(达姆施塔特工业大学计算机科学系泛在知识处理实验室(UKP实验室)与国家应用网络安全研究中心ATHENE);Department of Data Science and Artificial Intelligence (DSAI), Monash University(莫纳什大学数据科学与人工智能系(DSAI))
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Wenge Technology(文英科技)
Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration
The Chinese University of Hong Kong(香港中文大学);LLM Department, Tencent(腾讯语言模型部门);Mohamed bin Zayed University of Artificial Intelligence(马尔代夫人工智能大学);McGill University(麦吉尔大学);City University of Hong Kong(香港城市大学);The University of Edinburgh(爱丁堡大学)
Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration
Northeastern University(东北大学);University of Notre Dame(Notre Dame 大学);University of Waterloo(滑铁卢大学);Carnegie Mellon University(卡内基梅隆大学);Adobe(Adobe公司);Microsoft Research Asia(微软亚洲研究院)