代码大语言模型的无监督评估：基于往返正确性
Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
AI总结：
提出往返正确性（RTC）作为代码LLM的无监督评估方法，通过预测-反馈循环检查语义等价性，在广泛领域上评估代码合成与编辑，与现有基准强相关且无需人工标注。
摘要：
为了评估代码大语言模型（LLM），研究一直依赖于少量人工精心策划的基准，如HumanEval和MBPP，这些基准仅代表了现实世界软件领域的一小部分。在本工作中，我们引入了往返正确性（RTC）作为一种替代评估方法。RTC允许在更广泛的现实世界软件领域上评估代码LLM，而无需昂贵的人工策划。RTC基于这样一个想法：我们可以让模型做出预测（例如，用自然语言描述一些代码），将该预测反馈回去（例如，从预测的描述中合成代码），并检查这一往返过程是否导致与原始输入语义等价的代码。我们展示了如何利用RTC来评估代码合成和编辑。我们发现，RTC与现有窄领域代码合成基准上的模型性能强相关，同时使我们能够扩展到更广泛的领域和任务，这在以前没有昂贵的人工标注的情况下是不可能的。
英文摘要：
To evaluate code large language models (LLMs), research has relied on a few small manually curated benchmarks, such as HumanEval and MBPP, which represent a narrow part of the real-world software domains. In this work, we introduce round-trip correctness (RTC) as an alternative evaluation method. RTC allows Code LLM evaluation on a broader spectrum of real-world software domains without the need for costly human curation. RTC rests on the idea that we can ask a model to make a prediction (e.g., describe some code using natural language), feed that prediction back (e.g., synthesize code from the predicted description), and check if this round-trip leads to code that is semantically equivalent to the original input. We show how to employ RTC to evaluate code synthesis and editing. We find that RTC strongly correlates with model performance on existing narrow-domain code synthesis benchmarks while allowing us to expand to a much broader set of domains and tasks which was not previously possible without costly human annotations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
