arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.17833cs.SE
收藏

ArchBench:面向软件架构任务的生成式AI基准测试

ArchBench: Benchmarking Generative-AI for Software Architecture Tasks

Software Engineering Research Center (SERC) International Institute of Information Technology Hyderabad (IIIT-H)(印度信息技术研究所海德拉巴分校软件工程研究中心)

Bassam Adnan, Aviral Gupta, Sreemaee Akshathala, Karthik Vaidhyanathan

AI总结:

针对现有LLM基准多聚焦实现正确性、软件架构任务难统一评测的问题,研究提出首个统一基准平台ArchBench,含带标准化流程的CLI工具与带交互式排行榜的网页端,采用插件架构支持社区贡献,覆盖各类生成式AI软件工程方案,推动可复现研究。

摘要:

大语言模型(LLM)的基准测试已从代码片段级的函数生成发展到仓库级的问题解决,但绝大多数都聚焦于实现正确性。软件架构任务尽管在复杂系统的维护与演进中处于核心地位,但其定义仍不明确,且难以在不同模型间进行比较。我们提出了ArchBench,这是首个用于基准测试LLM软件架构任务能力的统一平台。ArchBench提供一款带有标准化流程的命令行工具,支持数据集下载、带轨迹记录的推理以及自动化评估,同时还配备了带有交互式排行榜的公开网页界面。该平台基于插件架构构建,每个任务都是一个独立模块,便于社区贡献新的架构任务和评估结果。我们宽泛地使用LLM这一术语,涵盖软件工程领域的生成式AI(GenAI)解决方案,既包括独立模型,也包括配备工具的基于LLM的编码智能体。命令行工具和网页平台均公开可用,以支持可复现研究以及架构基准测试的社区驱动式发展。

英文摘要:

Benchmarks for large language models (LLMs) have progressed from snippet-level function generation to repository-level issue resolution, yet they overwhelmingly target implementation correctness. Software architecture tasks remain under-specified and difficult to compare across models, despite their central role in maintaining and evolving complex systems. We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks. ArchBench provides a command-line tool with a standardized pipeline for dataset download, inference with trajectory logging, and automated evaluation, alongside a public web interface with an interactive leaderboard. The platform is built around a plugin architecture where each task is a self-contained module, making it straightforward for the community to contribute new architectural tasks and evaluation results. We use the term LLMs broadly to encompass generative AI (GenAI) solutions for software engineering, including both standalone models and LLM-based coding agents equipped with tools. Both the CLI tool and the web platform are openly available to support reproducible research and community-driven growth of architectural benchmarking.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑