arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16732cs.SEcs.AI
收藏

PyBench:在各类真实世界编码任务上评估 LLM 智能体

PyBench: Evaluating LLM Agent on various real-world coding tasks

School of Statistics, Renmin University of China(中国人民大学统计学院);ModelBest Inc.(模贝斯特公司)

Yaolun Zhang, Yinxu Pan, Yudong Wang, Jie Cai

AI总结:

本文提出覆盖五类真实任务的 PyBench,用于评估 LLM 智能体借助代码解释器完成日常 Python 编码的能力,并证明微调的 8B 模型 PyLlama3 可超越多个 33B/70B 模型。

摘要:

配备代码解释器的 LLM 智能体能够自动解决真实世界中的编码任务,例如数据分析和图像编辑。然而,现有基准主要要么聚焦于补全几行代码这类过于简单的任务,要么聚焦于仓库级别的极其复杂且特定的任务,二者都不能代表多样化的日常编码任务。为弥补这一空白,我们提出 PyBench,这一基准包含五大类真实世界任务,覆盖 10 多种文件类型。给定一个高层用户查询和相关文件,LLM 智能体需要通过代码解释器进行若干轮推理并执行 Python 代码,然后再给出正式响应以满足用户需求。成功完成 PyBench 中的任务,需要对各种 Python 包有扎实理解、出色的推理能力,以及吸收已执行代码反馈的能力。我们的评估表明,当前开源 LLM 在这些任务上表现吃力。因此,我们在四类数据集上进行分析和实验,证明 PyBench 需要综合能力。我们微调的 8B 规模模型 PyLlama3 在 PyBench 上取得了令人振奋的性能,超过了许多 33B 和 70B 规模模型。我们的基准、训练数据集和模型可在 https://github.com/Mercury7353/PyBench 获取。

英文摘要:

The LLM Agent, equipped with a code interpreter, is capable of automatically solving real-world coding tasks, such as data analysis and image editing. However, existing benchmarks primarily focus on either simplistic tasks, such as completing a few lines of code, or on extremely complex and specific tasks at the repository level, neither of which are representative of various daily coding tasks. To address this gap, we introduce \textbf{PyBench}, a benchmark encompassing five main categories of real-world tasks, covering more than 10 types of files. Given a high-level user query and related files, the LLM Agent needs to reason and execute Python code via a code interpreter for a few turns before making a formal response to fulfill the user's requirements. Successfully addressing tasks in PyBench demands a robust understanding of various Python packages, superior reasoning capabilities, and the ability to incorporate feedback from executed code. Our evaluations indicate that current open-source LLMs are struggling with these tasks. Hence, we conduct analysis and experiments on four kinds of datasets proving that comprehensive abilities are needed for PyBench. Our fine-tuned 8B size model: \textbf{PyLlama3} achieves an exciting performance on PyBench which surpasses many 33B and 70B size models. Our Benchmark, Training Dataset, and Model are available at: {https://github.com/Mercury7353/PyBench}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑