arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.07322cs.LG
收藏

ConceptSearch:面向利用大语言模型实现抽象与推理语料库(ARC)高效程序搜索的研究

ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)

TCS Research(TCS研究院)

Kartik Singhal, Gautam Shroff

AI总结:

针对大语言模型直接应用于ARC效果有限的问题,提出基于概念评分引导的ConceptSearch程序搜索算法,相比直接用GPT-4提示性能显著提升,且概念评分比汉明距离效率最高高30%。

摘要:

抽象与推理语料库(Abstraction and Reasoning Corpus,ARC)对人工智能构成重大挑战,它要求广泛的泛化能力和少样本学习能力,而当前包括大语言模型(LLM)在内的深度学习方法仍难以具备这些能力。尽管LLM在程序合成方面表现出色,但将其直接应用于ARC取得的成效有限。为解决这一问题,我们提出了ConceptSearch,这是一种新颖的函数搜索算法,它利用LLM生成程序,并采用基于概念的评分方法来高效引导搜索过程。与汉明距离这类简单的基于像素的指标不同,ConceptSearch根据程序捕捉输入输出示例中所体现的底层转换概念的能力来评估程序。我们探究了三种评分函数:汉明距离、基于CNN的评分函数以及基于LLM的自然语言评分函数。实验结果证明了ConceptSearch的有效性,与直接使用GPT-4提示相比,其性能得到了显著提升。此外,以达到正确解所需的迭代次数来衡量,我们提出的新型基于概念的评分方法比汉明距离的效率最高高出30%。这些发现凸显了将LLM驱动的程序搜索与基于概念的引导相结合,在解决ARC这类具有挑战性的泛化问题方面的潜力。

英文摘要:

The Abstraction and Reasoning Corpus (ARC) poses a significant challenge to artificial intelligence, demanding broad generalization and few-shot learning capabilities that remain elusive for current deep learning methods, including large language models (LLMs). While LLMs excel in program synthesis, their direct application to ARC yields limited success. To address this, we introduce ConceptSearch, a novel function-search algorithm that leverages LLMs for program generation and employs a concept-based scoring method to guide the search efficiently. Unlike simplistic pixel-based metrics like Hamming distance, ConceptSearch evaluates programs on their ability to capture the underlying transformation concept reflected in the input-output examples. We explore three scoring functions: Hamming distance, a CNN-based scoring function, and an LLM-based natural language scoring function. Experimental results demonstrate the effectiveness of ConceptSearch, achieving a significant performance improvement over direct prompting with GPT-4. Moreover, our novel concept-based scoring exhibits up to 30% greater efficiency compared to Hamming distance, measured in terms of the number of iterations required to reach the correct solution. These findings highlight the potential of LLM-driven program search when integrated with concept-based guidance for tackling challenging generalization problems like ARC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑