ARCANA:抽象与类比推理基准
ARCANA: A Benchmark for Abstraction and Analogical Reasoning
AI总结:
本研究提出远类比推理基准ARCANA,发现大型语言模型在该任务上性能远逊于人类,进而提出ConceptSIR框架,可使标准大型语言模型解决部分ARCANA难题,表明远类比推理仍是当前大模型的开放问题。
摘要:
类比推理被广泛认为是人类智能的核心组成部分。很少有基准被设计用于明确测试远类比推理——即对表面不同但具有深层关系或结构相似性的领域进行的推理。在本研究中,我们引入了名为Abstraction and Reasoning Corpus of ANAlogy(ARCANA)的类比推理基准。ARCANA的任务要求将现实世界对象或物理现象进行远类比映射,转换为基于网格的抽象变换,这与主要强调人类先天先验的知名ARC-AGI基准不同。我们通过一项人类研究验证了ARCANA,证明人类能够可靠地解决这些任务。相比之下,对一系列大型语言模型的评估显示,其性能与人类存在明显差距,暴露出当前模型仍难以克服的系统性失败模式。为解决这一差距,我们提出了基于概念滑移、内省和反思的推理框架ConceptSIR。实验表明,ConceptSIR使标准大型语言模型能够解决我们评估中直接提示未解决的若干具有挑战性的ARCANA任务。总体而言,我们的结果表明,远类比推理仍是当代大型语言模型的一个开放问题。所有材料可通过此http URL获取,包括基准和ConceptSIR在内的完整代码库可通过此https URL公开访问,所有ARCANA数据和任务实例可通过此https URL访问和探索。
英文摘要:
Analogical reasoning is widely considered a fundamental component of human intelligence. Few benchmarks are designed to explicitly test far analogical reasoning - reasoning between domains that are superficially dissimilar yet share deep relational or structural similarities. In this work, we introduce an analogical reasoning benchmark called Abstraction and Reasoning Corpus of ANAlogy (ARCANA). The tasks of ARCANA require performing far analogical mappings from real-world objects or physical phenomena to abstract grid-based transformations, different from the well-known ARC-AGI benchmark which primarily emphasized innate human priors. We validate ARCANA through a human study, demonstrating that humans solve these tasks reliably. In contrast, evaluations across a range of large language models show a distinct performance gap compared to humans, revealing systematic failure modes that remain challenging for current models. To address this gap, we propose ConceptSIR, a reasoning framework based on concept slippage, introspection, and reflection. Experiments demonstrate that ConceptSIR enables standard LLMs to solve several challenging ARCANA tasks that were unsolved by direct prompting in our evaluation. Overall, our results suggest that far analogical reasoning remains an open problem for contemporary LLMs. All the materials are available at this http URL. The complete codebase, including benchmark, and ConceptSIR, is publicly available at this https URL. All ARCANA data and task instances can be accessed and explored at this https URL.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。