arXiv 2506.026482025-09-28cs.AI

通过动态推理评估真正测评大型语言模型的流体智力

Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation

Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Harbin Institute of Technology（哈尔滨工业大学）；The University of Hong Kong（香港大学）

Yue Yang, MingKang Chen, Qihua Liu, Mengkang Hu, Qiguang Chen, Gengrui Zhang, Shuyue Hu, Guangtao Zhai, Yu Qiao, Yu Wang, Wenqi Shao, Ping Luo

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出基于分层认知框架的DRE-Bench动态推理基准，通过36项任务和多变体测试评估LLM流体智力，发现模型低级认知稳健但高级泛化不足。

摘要：

大型语言模型（LLM）的最新进展展现出令人瞩目的推理能力，反映出类似人类的思维。然而，LLM是否具备真正的流体智力（即在新情境中进行抽象推理并概括规则的能力）仍是一个悬而未决的问题。现有推理基准要么聚焦于特定领域知识（晶体智力），要么缺乏可解释性。为解决这些局限，我们提出DRE-Bench，这是一个基于分层认知框架的动态推理评估基准。DRE-Bench包含36项抽象推理任务，分布在四个认知层级；每项任务都具有多个动态变体，用于测试相同的潜在规则。该设计能够对流体智力进行细粒度、可解释且可靠的评估。我们评估了一系列最先进的LLM，包括通用LLM（GPT-4o、Claude 3.7）和推理LLM（o1、DeepSeek-R1、QwQ、Skywork-OR1）。实验结果表明，尽管大多数LLM在低级认知上取得了胜任且稳健的表现，但它们在高级认知方面存在困难，并且随着任务复杂性增加表现出有限的泛化能力。我们的发现凸显了当前LLM与真正类人流体智力之间的差距，并为系统追踪LLM推理进展提供了一条新路径。

英文摘要：

Recent advances in large language models (LLMs) have demonstrated impressive reasoning capacities that mirror human-like thinking. However, whether LLMs possess genuine fluid intelligence (i.e., the ability to reason abstractly and generalize rules in novel situations) remains an open question. Existing reasoning benchmarks either focus on domain-specific knowledge (crystallized intelligence) or lack interpretability. To address these limitations, we propose DRE-Bench, a dynamic reasoning evaluation benchmark grounded in a hierarchical cognitive framework. DRE-Bench consists of 36 abstract reasoning tasks organized across four cognitive levels, with each task featuring multiple dynamic variants that test the same underlying latent rule. This design enables fine-grained, interpretable, and reliable assessments of fluid intelligence. We evaluate a range of state-of-the-art LLMs, including both general LLMs (GPT-4o, Claude 3.7) and reasoning LLMs (o1, DeepSeek-R1, QwQ, Skywork-OR1). Experimental results reveal that although most LLMs achieve competent and robust performance in low-level cognition, they struggle with high-level cognition and exhibit limited generalization as task complexity grows. Our findings highlight the gap between current LLMs and true human-like fluid intelligence and offer a new path for systematically tracking reasoning progress in LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。