大型语言模型的推理能力：对抽象与推理语料库的深入分析
Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus
AI总结：
本文提出基于思维语言假说的过程导向评估方法，利用ARC基准分析LLMs在逻辑连贯性、组合性和生成性上的推理能力，发现其仍显著落后于人类水平，为发展人类级AI推理提供了新视角。
摘要：
现有评估大型语言模型（LLMs）推理能力的方法主要以结果为中心，难以全面评估推理过程。我们提出一种新方法，利用抽象与推理语料库（ARC）基准，以过程为中心的方式评估LLMs的推理和上下文理解能力，重点关注思维语言假说（LoTH）中的三个关键组成部分：逻辑连贯性、组合性和生成性。我们精心设计的实验表明，尽管LLMs展现出一定的推理能力，但在这三个方面仍显著落后于人类水平的推理。本文的主要贡献在于引入LoTH视角，为评估推理过程提供了一种传统以结果为导向的方法无法捕捉的方式，从而为人工智能系统中人类水平推理的发展提供了新的见解。
英文摘要：
The existing methods for evaluating the inference abilities of Large Language Models (LLMs) have been predominantly results-centric, making it challenging to assess the inference process comprehensively. We introduce a novel approach using the Abstraction and Reasoning Corpus (ARC) benchmark to evaluate the inference and contextual understanding abilities of LLMs in a process-centric manner, focusing on three key components from the Language of Thought Hypothesis (LoTH): Logical Coherence, Compositionality, and Productivity. Our carefully designed experiments reveal that while LLMs demonstrate some inference capabilities, they still significantly lag behind human-level reasoning in these three aspects. The main contribution of this paper lies in introducing the LoTH perspective, which provides a method for evaluating the reasoning process that conventional results-oriented approaches fail to capture, thereby offering new insights into the development of human-level reasoning in artificial intelligence systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
