抽象与推理语料库(Abstraction and Reasoning Corpus,ARC)对人工智能构成重大挑战,它要求广泛的泛化能力和少样本学习能力,而当前包括大语言模型(LLM)在内的深度学习方法仍难以具备这些能力。尽管LLM在程序合成方面表现出色,但将其直接应用于ARC取得的成效有限。为解决这一问题,我们提出了ConceptSearch,这是一种新颖的函数搜索算法,它利用LLM生成程序,并采用基于概念的评分方法来高效引导搜索过程。与汉明距离这类简单的基于像素的指标不同,ConceptSearch根据程序捕捉输入输出示例中所体现的底层转换概念的能力来评估程序。我们探究了三种评分函数:汉明距离、基于CNN的评分函数以及基于LLM的自然语言评分函数。实验结果证明了ConceptSearch的有效性,与直接使用GPT-4提示相比,其性能得到了显著提升。此外,以达到正确解所需的迭代次数来衡量,我们提出的新型基于概念的评分方法比汉明距离的效率最高高出30%。这些发现凸显了将LLM驱动的程序搜索与基于概念的引导相结合,在解决ARC这类具有挑战性的泛化问题方面的潜力。
英文摘要:
The Abstraction and Reasoning Corpus (ARC) poses a significant challenge to artificial intelligence, demanding broad generalization and few-shot learning capabilities that remain elusive for current deep learning methods, including large language models (LLMs). While LLMs excel in program synthesis, their direct application to ARC yields limited success. To address this, we introduce ConceptSearch, a novel function-search algorithm that leverages LLMs for program generation and employs a concept-based scoring method to guide the search efficiently. Unlike simplistic pixel-based metrics like Hamming distance, ConceptSearch evaluates programs on their ability to capture the underlying transformation concept reflected in the input-output examples. We explore three scoring functions: Hamming distance, a CNN-based scoring function, and an LLM-based natural language scoring function. Experimental results demonstrate the effectiveness of ConceptSearch, achieving a significant performance improvement over direct prompting with GPT-4. Moreover, our novel concept-based scoring exhibits up to 30% greater efficiency compared to Hamming distance, measured in terms of the number of iterations required to reach the correct solution. These findings highlight the potential of LLM-driven program search when integrated with concept-based guidance for tackling challenging generalization problems like ARC.
The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning
TIB - Leibniz Information Centre for Science and Technology(TIB - 科学技术信息研究中心);L3S Research Center, Leibniz University of Hannover(L3S研究所以汉诺威莱布尼茨大学);Dresden University of Technology, Amazon (work done outside of Amazon)(德累斯顿技术大学,亚马逊(非亚马逊工作))
Vector Symbolic Algebras for the Abstraction and Reasoning Corpus
Centre for Theoretical Neuroscience(理论神经科学中心);David R. Cheriton School of Computer Science(大卫·R·切里顿计算机科学学院);University of Waterloo(滑铁库大学);Department of Philosophy(哲学系);Department of Systems Design Engineering(系统设计工程系)