arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.08316cs.CLcs.AIcs.LGcs.PLcs.SE
收藏

示例编程是否已被大语言模型解决?

Is Programming by Example solved by LLMs?

Cornell University(康奈尔大学)

Wen-Ding Li, Kevin Ellis

AI总结:

本文通过列表、字符串和图形编程领域的实验,发现预训练大语言模型在示例编程(PBE)上效果有限,但经微调后在分布内任务上性能大幅提升,并分析了成功与失败原因及分布外泛化改进方向。

摘要:

示例编程(Programming-by-Examples, PBE)旨在根据输入-输出示例生成算法。这类系统在实践和理论上都具有重要意义:从终端用户角度看,它们已部署给数百万用户;从人工智能角度看,PBE 对应一种非常通用的少样本归纳推理形式。鉴于大语言模型(LLMs)在代码生成任务中取得的成功,本文研究 LLMs 在多大程度上可以被认为已经“解决”了 PBE。我们在列表和字符串等经典领域,以及一个在典型预训练数据中未被充分代表的罕见图形编程领域上进行了实验。我们发现,预训练模型在 PBE 上效果不佳,但如果测试问题处于分布内,则可以通过微调大幅提升性能。我们从实证角度分析了这些模型成功和失败的原因,并朝着理解如何实现更好的分布外泛化迈出了步伐。综合来看,这些结果表明,LLMs 在解决典型 PBE 任务套件方面取得了显著进展,有望提高 PBE 系统的灵活性和适用性,同时也揭示了 LLMs 仍然存在不足的方面。

英文摘要:

Programming-by-Examples (PBE) aims to generate an algorithm from input-output examples. Such systems are practically and theoretically important: from an end-user perspective, they are deployed to millions of people, and from an AI perspective, PBE corresponds to a very general form of few-shot inductive inference. Given the success of Large Language Models (LLMs) in code-generation tasks, we investigate here the extent to which LLMs can be said to have "solved" PBE. We experiment on classic domains such as lists and strings, and an uncommon graphics programming domain not well represented in typical pretraining data. We find that pretrained models are not effective at PBE, but that they can be fine-tuned for much higher performance, provided the test problems are in-distribution. We analyze empirically what causes these models to succeed and fail, and take steps toward understanding how to achieve better out-of-distribution generalization. Collectively these results suggest that LLMs make strong progress toward solving the typical suite of PBE tasks, potentially increasing the flexibility and applicability of PBE systems, while also identifying ways in which LLMs still fall short.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑