arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.11206cs.CLcs.AIcs.LG
收藏

捕捉 ARC Challenge 的抽象火花

Capturing Sparks of Abstraction for the ARC Challenge

Red Dragon AI(红龙人工智能公司)

Martin Andrews

AI总结:

针对 ARC Challenge 突破 60% 准确率的瓶颈,本研究提出从完整代码解决方案出发,利用 LLM 在多层级抽象上解释任务,提取可用于本地 LLM 下游任务的“抽象火花”,并将相关框架与数据开源。

摘要:

近期在解决 ARC Challenge 问题上取得了显著进展。然而,似乎需要新技术才能突破 60% 的准确率。即使是商业大型语言模型(LLM)也难以“理解”许多问题(在给定输入和输出网格时),这使得通过 LLM 主导的程序搜索来发现解决方案有些徒劳。在本研究中,尝试从更强的起点进行 LLM“理解”:为 LLM 提供任务的完整代码解决方案,然后要求其在不同抽象层级解释任务的解决方式。具体而言,向 LLM 提供在 arc-dsl-llm(Hodel 的 arc-dsl 的 LLM 可读版本)中实现的代码解决方案,以获得:(a) 带注释的代码;(b) 重构为可重用功能块的代码;(c) 问题解决步骤;以及 (d) 高级问题解决策略。我们证明,可以从 LLM 输出中提取“抽象火花”——其形式可用于有资格参加 ARC Prize 的本地 LLM 的下游任务。arc-dsl-llm DSL 框架(包含重新设计的解决方案)和 Gemini LLM 生成的数据(连同生成代码)均已开源。

英文摘要:

Excellent progress has been made recently in solving ARC Challenge problems. However, it seems that new techniques may be required to push beyond 60% accuracy. Even commercial Large Language Models (LLMs) struggle to 'understand' many of the problems (when given the input and output grids), which makes discovering solutions by LLM-lead program search somewhat futile. In this work, LLM 'understanding' is attempted from a stronger starting position : An LLM is given complete solutions to tasks in code, and then asked to explain how the task is being solved at various levels of abstraction. Specifically, the LLM was given code solutions implemented in arc-dsl-llm (an LLM-legible version of Hodel's arc-dsl to obtain: (a) commented code; (b) code refactored into reusable functional chunks; (c) problem solution steps; and (d) high-level problem-solving tactics. We demonstrate that 'Sparks of Abstraction' can be extracted from the LLM output - in a form that could be used in downstream tasks with Local LLMs eligible to enter the ARC Prize. Both the arc-dsl-llm DSL framework (with the re-engineered solutions) and the Gemini LLM-generated data (along with the generation code) are made Open Source.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑