ECCO:我们能否在不牺牲功能正确性的前提下提升模型生成代码的效率?
ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?
AI总结:
针对大语言模型生成代码时效率与正确性难以兼顾、效率基准测试不可靠的问题,本文提出可复现基准ECCO,测试三类现有方法后发现执行信息助于保正确性、自然语言反馈更能提效率,为后续高效代码生成研究提供支持。
摘要:
尽管大语言模型(LLMs)在生成功能正确的程序方面已取得巨大成功,但让模型在生成高效解决方案的同时保证正确性仍是一项挑战。此外,对于Python这类流行的解释型语言,不同硬件规格下代码效率基准测试的不可靠性也是一大障碍。本文提出ECCO,这是一个可复现的基准测试框架,通过两种范式评估程序效率:基于自然语言(NL)的代码生成和基于历史的代码编辑。我们在ECCO上适配并深入研究了现有三种最具潜力的基于LLM的方法:上下文学习、结合执行反馈或自然语言反馈的迭代优化,以及基于执行和编辑历史的微调。实验发现,大多数方法会降低功能正确性,仅能适度提升程序效率;而加入执行信息通常有助于维持功能正确性,自然语言反馈则对效率的提升更显著。我们发布该基准以支持未来基于LLM的高效代码生成研究。
英文摘要:
Although large language models (LLMs) have been largely successful in generating functionally correct programs, conditioning models to produce efficient solutions while ensuring correctness remains a challenge. Further, unreliability in benchmarking code efficiency is a hurdle across varying hardware specifications for popular interpreted languages such as Python. In this paper, we present ECCO, a reproducible benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. On ECCO, we adapt and thoroughly investigate the three most promising existing LLM-based approaches: in-context learning, iterative refinement with execution or NL feedback, and fine-tuning conditioned on execution and editing history. While most methods degrade functional correctness and moderately increase program efficiency, we find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency. We release our benchmark to support future work on LLM-based generation of efficient code.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。