arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.09112cs.AI
收藏

一个小型系统用于自回归程序合成,以实现受控实验

A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation

Apple(苹果公司)

Russ Webb, Jason Ramapuram

AI总结:

Cadmus系统通过低成本模型实现复杂程序合成任务,展示其在整数算术任务中的优越性能并揭示大模型的局限性。

摘要:

通常,研究人员通过大型语言模型(LLMs)研究程序合成,这会引入诸如知道什么在分布内或外、理解微调效果、理解标记化的影响以及对计算和存储的更高需求等问题。我们提出了一个名为Cadmus的系统,包括一个整数虚拟机(VM)、一个由各种任务的真实程序组成的数据集,以及一个经过不到200美元计算成本训练的自回归转换器模型。该系统可用于研究程序完成、分布外表示、归纳推理和指令遵循,研究者可以在具有有效且经济的细粒度训练分布控制能力和模型检查和仪器化的环境中进行研究。在复杂推理任务上工作的小型模型使能够对可能在更大模型上难以承受的仪器化和研究进行探索。为了证明这些任务足够复杂以引起兴趣,我们展示了这些Cadmus模型在我们的领域特定语言(DSL)中完成正确整数算术程序的任务上优于GPT-5(在该任务上达到100%的准确率,而GPT-5为95%的准确率),同时提供了数据集与问题关系的透明度。我们还展示了GPT-5在解决相同任务时会引入未知的先验知识,这表明了一个阻碍某些研究的混淆因素,这些研究需要完全理解训练集与任务之间的关系。

英文摘要:

What research can be pursued with small models trained to complete true programs? Typically, researchers study program synthesis via large language models (LLMs) which introduce issues such as knowing what is in or out of distribution, understanding fine-tuning effects, understanding the effects of tokenization, and higher demand on compute and storage to carry out experiments. We present a system called Cadmus which includes an integer virtual machine (VM), a dataset composed of true programs of diverse tasks, and an autoregressive transformer model that is trained for under \$200 of compute cost. The system can be used to study program completion, out-of-distribution representations, inductive reasoning, and instruction following in a setting where researchers have effective and affordable fine-grained control of the training distribution and the ability to inspect and instrument models. Smaller models working on complex reasoning tasks enable instrumentation and investigations that may be prohibitively expensive on larger models. To demonstrate that these tasks are complex enough to be of interest, we show that these Cadmus models outperform GPT-5 (by achieving 100\% accuracy while GPT-5 has 95\% accuracy) even on a simple task of completing correct, integer arithmetic programs in our domain-specific language (DSL) while providing transparency into the dataset's relationship to the problem. We also show that GPT-5 brings unknown priors into its reasoning process when solving the same tasks, demonstrating a confounding factor that prevents the use of large-scale LLMs for some investigations where the training set relationship to the task needs to be fully understood.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑