CogMath:从人类认知视角评估大语言模型的真实数学能力
CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective
AI总结:
本文提出 CogMath 评估框架,从人类认知视角将数学推理分为三阶段并设计 9 个细粒度维度,通过多智能体系统生成询问,揭示主流 LLMs 数学能力被高估 30%-40%,并定位其具体优势与不足。
摘要:
尽管大语言模型(LLMs)在解决复杂数学任务方面展现出潜力,但现有评估范式仅依赖整体答案准确率的粗粒度度量,不足以评估其真实能力。本文提出 CogMath,从人类认知视角全面评估 LLMs 的数学能力。具体而言,受心理学理论启发,CogMath 将人类推理过程形式化为三个阶段:问题理解、问题求解和解答总结。在这些阶段中,我们考察数值计算、知识和反事实等视角,并设计了共 9 个细粒度评估维度。在每个维度中,我们开发了一个“Inquiry-Judge-Reference”多智能体系统,用于生成评估 LLMs 在该维度掌握程度的询问。只有当 LLM 在 9 个维度的所有询问中均表现优异时,才被认为真正掌握了一个问题。通过在三个基准上应用 CogMath,我们揭示出 7 个主流 LLMs 的数学能力被高估了 30%-40%。此外,我们定位了它们在特定阶段/维度上的优势与不足,为进一步增强其推理能力提供了深入见解。
英文摘要:
Although large language models (LLMs) show promise in solving complex mathematical tasks, existing evaluation paradigms rely solely on a coarse measure of overall answer accuracy, which are insufficient for assessing their authentic capabilities. In this paper, we propose \textbf{CogMath}, which comprehensively assesses LLMs' mathematical abilities through the lens of human cognition. Specifically, inspired by psychological theories, CogMath formalizes human reasoning process into 3 stages: \emph{problem comprehension}, \emph{problem solving}, and \emph{solution summarization}. Within these stages, we investigate perspectives such as numerical calculation, knowledge, and counterfactuals, and design a total of 9 fine-grained evaluation dimensions. In each dimension, we develop an ``\emph{Inquiry}-\emph{Judge}-\emph{Reference}'' multi-agent system to generate inquiries that assess LLMs' mastery from this dimension. An LLM is considered to truly master a problem only when excelling in all inquiries from the 9 dimensions. By applying CogMath on three benchmarks, we reveal that the mathematical capabilities of 7 mainstream LLMs are overestimated by 30\%-40\%. Moreover, we locate their strengths and weaknesses across specific stages/dimensions, offering in-depth insights to further enhance their reasoning abilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。