EDIT-Bench：评估大语言模型（LLM）执行真实世界指令式代码编辑的能力
EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
AI总结：
针对AI编码助手的指令式代码编辑能力缺乏真实世界基准的问题，本文提出基于真实场景构建的EDIT-Bench基准，评估40个LLM后发现其具有挑战性且上下文对性能影响显著，为该能力评估提供了有效工具。
摘要：
指令式代码编辑是指LLM根据用户指令直接修改开发者的现有代码，这一模式正成为AI编码助手广泛使用的交互方式。然而，直接评估该能力的基准很少，现有数据集通常依赖人工来源。我们引入EDIT-Bench，一个基于真实世界使用场景的LLM代码编辑能力评估基准，即收集真实环境中的用户指令和代码上下文。EDIT-Bench包含540个问题、多种自然语言和编程语言，以及从错误修复到功能添加的各类真实世界用例。它引入了依赖上下文的问题，要求模型除理解用户指令外，还要理解代码上下文、高亮代码和光标位置。我们评估了40个不同的LLM，发现EDIT-Bench是一组具有挑战性的问题，只有1个模型得分超过60%。我们还发现模型性能因用户指令类别而异，且不同水平的上下文信息对任务成功率影响显著，性能差异高达11%，这表明用真实上下文进行评估的重要性。
英文摘要：
Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. However, few benchmarks directly evaluate this capability and current datasets often rely on artificial sources. We introduce EDIT-Bench, a benchmark for evaluating LLM code editing capabilities grounded in real-world usage, i.e., user instructions and code contexts collected in the wild. EDIT-Bench comprises of 540 problems, multiple natural and programming languages, and a diverse set of real-world use cases, ranging from resolving errors to adding features. EDIT-Bench introduces context-dependent problems that require the model to understand code context, highlighted code, and cursor position in addition to the user instruction. We evaluate 40 diverse LLMs and observe that EDIT-Bench is a challenging set of problems where only 1 model scores over 60%. We find that model performance varies across different categories of user instructions. Further, we find that varying levels of contextual information greatly affect task success rate, with performance varying up to 11%, indicating the importance of evaluating with realistic context.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
