CodeIt：基于优先后见重放的自改进语言模型
CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay
AI总结：
CodeIt将ARC视为示例编程问题，通过程序采样与后见重标记结合优先经验重放实现语言模型自改进，是首个扩展到完整ARC数据集的神经符号方法，解决15%任务并达到最先进性能。
摘要：
大型语言模型正越来越多地解决通常被认为需要人类级推理能力的任务。然而，这些模型在通用智能基准测试（如抽象与推理语料库（ARC））上仍然表现很差。在本文中，我们将ARC视为一个示例编程问题，并提出了一种新颖且可扩展的语言模型自改进方法，称为代码迭代（CodeIt）。我们的方法在以下两个阶段之间迭代：1）程序采样与后见重标记，以及2）从优先经验重放中学习。通过将一个回合的目标（即给定输入的目标程序输出）重标记为采样程序实际产生的输出，我们的方法有效应对了程序合成中奖励极度稀疏的问题。将CodeIt应用于ARC数据集，我们证明优先后见重放与预训练和数据增强相结合，能够实现成功的跨任务泛化。CodeIt是第一个扩展到完整ARC评估数据集的神经符号方法。我们的方法解决了15%的ARC评估任务，达到了最先进的性能，并优于现有的神经和符号基线。我们的代码可在 https://github.com/Qualcomm-AI-research/codeit 获取。
英文摘要：
Large language models are increasingly solving tasks that are commonly believed to require human-level reasoning ability. However, these models still perform very poorly on benchmarks of general intelligence such as the Abstraction and Reasoning Corpus (ARC). In this paper, we approach ARC as a programming-by-examples problem, and introduce a novel and scalable method for language model self-improvement called Code Iteration (CodeIt). Our method iterates between 1) program sampling and hindsight relabeling, and 2) learning from prioritized experience replay. By relabeling the goal of an episode (i.e., the target program output given input) to the realized output produced by the sampled program, our method effectively deals with the extreme sparsity of rewards in program synthesis. Applying CodeIt to the ARC dataset, we demonstrate that prioritized hindsight replay, along with pre-training and data-augmentation, leads to successful inter-task generalization. CodeIt is the first neuro-symbolic approach that scales to the full ARC evaluation dataset. Our method solves 15% of ARC evaluation tasks, achieving state-of-the-art performance and outperforming existing neural and symbolic baselines. Our code is available at https://github.com/Qualcomm-AI-research/codeit .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
