arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.06111cs.SEcs.AIcs.LG
收藏

CSR-Bench:计算机科学研究仓库部署中LLM智能体的基准测试

CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories

University of California, Los Angeles(加州大学洛杉矶分校);Amazon Web Services(亚马逊网络服务)

Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, Wei Wang

AI总结:

针对计算机科学研究仓库部署需求,提出CSR-Bench基准测试与CSR-Agents框架,多维度评估LLMs处理复杂代码开发任务的能力,初步验证其可优化部署流程并提升开发者生产力。

摘要:

计算机科学研究项目日益复杂,需要更有效的代码仓库部署工具。大型语言模型(LLMs)如Anthropic Claude和Meta Llama在计算机科学研究的多个领域取得显著进展,包括自动化各类软件工程任务。为评估LLMs处理研究项目复杂代码开发任务(尤其针对NLP/CV/AI/ML/DM主题)的效果,我们引入CSR-Bench——计算机科学研究项目基准测试。该基准从准确性、效率、部署脚本质量等方面评估LLMs,旨在探索其自主开展计算机科学研究的潜力。初步结果显示,LLM智能体可显著优化仓库部署流程,提升开发者生产力并改善开发流程管理。

英文摘要:

The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑