arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16801cs.CL
收藏

RES-Q:在仓库尺度上评估代码编辑大型语言模型系统

RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale

Northeastern University(东北大学);Qurrent AI

Beck LaBash, August Rosedale, Alex Reents, Lucas Negritto, Colin Wiel

AI总结:

针对大型语言模型系统在代码仓库编辑任务中缺乏稳健评估工具的问题,提出基于自然语言指令的基准 RES-Q,包含 100 个真实 GitHub 提交任务,全面评估系统的指令解释、信息检索与代码编辑能力,揭示了传统基准无法体现的模型差异。

摘要:

大型语言模型(LLMs)的指令遵循能力催生了一类基于 LLM 的系统,它们能够处理诸如对大型代码仓库进行编辑等复杂任务。由于 LLM 行为对提示变化的高度敏感性和不可预测性,需要稳健的评估工具来推动这些系统的未来迭代。我们提出了 RES-Q,这是一个基于自然语言指令的基准,用于评估仓库编辑系统(Repository Editing Systems),它包含 100 个源自真实 GitHub 提交的手工制作的仓库编辑任务。给定一条编辑指令和一个代码仓库,RES-Q 评估 LLM 系统解释指令、浏览仓库以收集相关信息,以及构建满足指定标准的适当编辑的能力。我们认为,以这种方式评估 LLMs 解决了传统基准的问题,并对模型的能力提供了更全面的评估。我们在基于我们的语言智能体开发软件 Qurrent OS 构建的仓库编辑系统中,将各种最先进的 LLMs 作为语言智能体进行了评估。尽管 Claude Sonnet 3.5 和 GPT-4o 在 HumanEval 上的 pass@1 性能差异仅为 1%,但我们发现 Claude Sonnet 3.5 在 RES-Q 上的 pass@1 比 GPT-4o 高出 12%,这表明随着传统基准接近饱和,RES-Q 具备区分模型能力的能力。我们进一步研究了 token 效率、与现有基准的性能关系,以及闭源与开源 LLMs 之间有趣的差异。代码和数据集可在 https://github.com/Qurrent-AI/RES-Q 获取。

英文摘要:

The instruction-following ability of Large Language Models (LLMs) has cultivated a class of LLM-based systems capable of approaching complex tasks such as making edits to large code repositories. Due to the high sensitivity and unpredictability of LLM behavior in response to changes in prompting, robust evaluation tools are needed to drive future iteration of these systems. We propose RES-Q, a natural language instruction-based benchmark for evaluating $\textbf{R}$epository $\textbf{E}$diting $\textbf{S}$ystems, which consists of 100 handcrafted repository editing tasks derived from real GitHub commits. Given an edit instruction and a code repository, RES-Q evaluates an LLM system's ability to interpret the instruction, navigate the repository to gather relevant information, and construct an appropriate edit that satisfies the specified criteria. We argue that evaluating LLMs in this way addresses issues with traditional benchmarks and provides a more holistic assessment of a model's abilities. We evaluate various state-of-the-art LLMs as language agents in a repository-editing system built on Qurrent OS, our language agent development software. Despite their 1% pass@1 performance difference on HumanEval, we find Claude Sonnet 3.5 outperforms GPT-4o by 12% pass@1 on RES-Q, indicating RES-Q's capacity to differentiate model capability as traditional benchmarks approach saturation. We further investigate token efficiency, performance relationships with existing benchmarks, and interesting disparities between closed and open-source LLMs. Code and dataset are available at https://github.com/Qurrent-AI/RES-Q.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑