arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07079cs.SEcs.CL
收藏

对大型语言模型在软件工程任务中的综合评估:一个多任务基准测试

Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark

Go Frendi Gunawan, Mukhlis Amien

AI总结:

本文评估了11种先进LLMs在五个软件工程任务中的表现,发现模型在完成时间、工具效率和成本上存在显著差异,同时揭示了两种低效模式,并展示了编码任务的成功率与研究任务的挑战性差异。

摘要:

大型语言模型(LLMs)在软件工程领域展现出了显著的能力,但全面覆盖多样软件工程活动的基准测试仍然有限。我们对11种最先进的LLMs在五个代表性的软件工程任务上进行了多任务评估:bug修复、功能开发、代码重构、技术文案写作和技术综合。我们的自动化验证框架测量了输出质量和完成效率。关键发现显示:(1)达到相同完美分数的模型在完成时间上存在22倍的差异,工具效率存在49倍的差异,估计成本存在53倍的差异;(2)工具使用频率与成功无相关性(r=0.077,p=0.575)-一个模型使用了917次工具调用,而另一个模型在解决相同任务时仅使用了3次调用;(3)我们识别出两种不同的低效模式:循环低效和推理低效;(4)编码任务实现了100%的成功率,而研究任务则更具挑战性(90.9%)。我们释放了所有实验数据、验证脚本和分析代码以确保完全可复现。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering, yet comprehensive benchmarks covering diverse SE activities remain limited. We present a multi-task evaluation of 11 state-of-the-art LLMs across five representative software engineering tasks: bug fixing, feature development, code refactoring, technical copywriting, and research synthesis. Our automated verification framework measures both output quality and completion efficiency. Key findings reveal that (1) models achieving identical perfect scores exhibit 22x variation in completion time, 49x variation in tool efficiency, and 53x variation in estimated cost; (2) tool usage frequency shows no correlation with success (r = 0.077, p = 0.575) - one model used 917 tool calls while another solved the same task with 3 calls; (3) we identify two distinct inefficiency patterns: loop inefficiency and inference inefficiency; and (4) coding tasks achieve 100 percent success while research tasks present greater challenges (90.9 percent). We release all experimental data, verification scripts, and analysis code for full reproducibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑