arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.13243cs.LG
收藏

大语言模型推理的全面评估:从单模型到多智能体范式

A Comprehensive Evaluation of LLM Reasoning: From Single-Model to Multi-Agent Paradigms

Harbin Institute of Technology(哈尔滨工业大学)

Yapeng Li, Jiakuo Yu, Zhixin Liu, Xinnan Liu, Jing Yu, Songze Li, Tonghua Su

AI总结:

本研究全面评估了LLM推理范式,包括单模型和多智能体系统,通过新基准测试揭示了不同范式在成本与准确率之间的权衡。

摘要:

大语言模型(LLMs)正越来越多地被部署为推理系统,其中推理范式——如链式思考(CoT)和多智能体系统(MAS)——在其中扮演着关键角色,但它们的相对有效性及成本-准确率权衡仍缺乏深入理解。在本工作中,我们对推理范式进行了全面且统一的评估,涵盖了直接的单模型生成、增强CoT的单模型推理以及具有代表性的MAS工作流程,通过多样化的封闭式基准测试套件来表征其推理性能。除了整体性能外,我们通过定向角色隔离分析来探测MAS中的角色特定能力需求,并分析成本-准确率权衡,以确定哪些MAS工作流程在成本与准确率之间提供了有利的平衡,哪些则因边际收益而带来过度开销。我们进一步引入了MIMeBench,一个新的开放性基准测试,旨在针对两个基础但尚未被充分探索的语义能力——语义抽象和对比性辨别——从而提供了一个超越封闭式准确率的评估轴线,使能够对语义能力进行细致评估,而现有基准测试难以捕捉到。我们的结果表明,增加的结构复杂性并不总是导致推理性能的提升,其收益高度依赖于推理范式的性质和适用性。代码已发布在https://gitcode.com/HIT1920/OpenLLMBench。

英文摘要:

Large Language Models (LLMs) are increasingly deployed as reasoning systems, where reasoning paradigms - such as Chain-of-Thought (CoT) and multi-agent systems (MAS) - play a critical role, yet their relative effectiveness and cost-accuracy trade-offs remain poorly understood. In this work, we conduct a comprehensive and unified evaluation of reasoning paradigms, spanning direct single-model generation, CoT-augmented single-model reasoning, and representative MAS workflows, characterizing their reasoning performance across a diverse suite of closed-form benchmarks. Beyond overall performance, we probe role-specific capability demands in MAS using targeted role isolation analyses, and analyze cost-accuracy trade-offs to identify which MAS workflows offer a favorable balance between cost and accuracy, and which incur prohibitive overhead for marginal gains. We further introduce MIMeBench, a new open-ended benchmark that targets two foundational yet underexplored semantic capabilities - semantic abstraction and contrastive discrimination - thereby providing an alternative evaluation axis beyond closed-form accuracy and enabling fine-grained assessment of semantic competence that is difficult to capture with existing benchmarks. Our results show that increased structural complexity does not consistently lead to improved reasoning performance, with its benefits being highly dependent on the properties and suitability of the reasoning paradigm itself. The codes are released at https://gitcode.com/HIT1920/OpenLLMBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑