arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.02810cs.CLcs.AIcs.LG
收藏

大型语言模型中复杂推理的生成式评估

Generative Evaluation of Complex Reasoning in Large Language Models

Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Computer Science Department, Stanford University(斯坦福大学计算机科学系)

Haowei Lin, Xiangyu Wang, Ruilin Yan, Baizhou Huang, Haotian Ye, Jianhua Zhu, Zihao Wang, James Zou, Jianzhu Ma, Yitao Liang

AI总结:

提出生成式评估框架 KUMO,结合 LLM 与符号引擎动态生成多轮推理任务,评估 23 个模型在 5000 任务上的表现,发现许多模型在简单任务上超越大学生,推理优化模型在复杂任务上达到大学水平。

摘要:

随着强大的大型语言模型(LLMs)展现出超人类的推理能力,一个关键问题随之浮现:LLMs 是真正在进行推理,还是仅仅从其庞大的、从网络抓取的训练数据中回忆答案?公开的基准测试一旦被纳入后续的 LLM 训练集,就不可避免地受到污染,从而削弱其作为可靠评估的可信度。为解决这一问题,我们提出了 KUMO,一个专门用于评估 LLMs 推理能力的生成式评估框架。KUMO 将 LLMs 与符号引擎协同结合,动态生成多样化的、多轮推理任务,这些任务具有部分可观测性且难度可调。通过自动化流程,KUMO 在开放领域持续生成新颖任务,迫使模型展示真正的泛化能力而非记忆能力。我们在 KUMO 生成的 100 个领域的 5,000 个任务上评估了 23 个最先进的 LLMs,并将其推理能力与大学生进行了对比。我们的研究结果显示,许多 LLMs 在简单推理任务上已超越大学水平,而经过推理优化的 LLMs 在复杂推理挑战中达到了大学水平。此外,LLMs 在 KUMO 任务上的表现与新发布的真实世界推理基准的结果高度相关,这凸显了 KUMO 作为评估 LLMs 真实推理能力的稳健且持久的工具的价值。

英文摘要:

With powerful large language models (LLMs) demonstrating superhuman reasoning capabilities, a critical question arises: Do LLMs genuinely reason, or do they merely recall answers from their extensive, web-scraped training datasets? Publicly released benchmarks inevitably become contaminated once incorporated into subsequent LLM training sets, undermining their reliability as faithful assessments. To address this, we introduce KUMO, a generative evaluation framework designed specifically for assessing reasoning in LLMs. KUMO synergistically combines LLMs with symbolic engines to dynamically produce diverse, multi-turn reasoning tasks that are partially observable and adjustable in difficulty. Through an automated pipeline, KUMO continuously generates novel tasks across open-ended domains, compelling models to demonstrate genuine generalization rather than memorization. We evaluated 23 state-of-the-art LLMs on 5,000 tasks across 100 domains created by KUMO, benchmarking their reasoning abilities against university students. Our findings reveal that many LLMs have outperformed university-level performance on easy reasoning tasks, and reasoning-scaled LLMs reach university-level performance on complex reasoning challenges. Moreover, LLM performance on KUMO tasks correlates strongly with results on newly released real-world reasoning benchmarks, underscoring KUMO's value as a robust, enduring assessment tool for genuine LLM reasoning capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑