arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.18525cs.SEcs.LG
收藏

像素编程:计算机使用智能体能完成软件工程工作吗?

Programming with Pixels: Can Computer-Use Agents do Software Engineering?

Carnegie Mellon University(卡内基梅隆大学)

Pranjal Aggarwal, Sean Welleck

AI总结:

该研究提出面向软件工程的计算机使用环境PwP及PwP-Bench基准,评估发现纯视觉下通用CUA表现弱于专用编程智能体,赋予文件编辑、bash操作等API访问权限后性能可接近甚至进一步提升,为CUA改进指明方向。

摘要:

计算机使用智能体(CUA)有望执行各类通用任务,但当前评估主要聚焦于简单场景,因此这类通用智能体能否自动化完成软件工程(SWE)这类更复杂的专业工作仍不明确。为探究该问题,我们提出了$\texttt{Programming with Pixels}$(PwP,像素编程)——首个面向软件工程的综合计算机使用环境,智能体可在其中通过视觉控制集成开发环境(IDE)执行各类软件工程任务。为实现全面评估,我们还推出了\texttt{PwP-Bench}基准,包含15项既有及新增的软件工程任务,覆盖多模态、多编程语言与多技能维度。我们对当前最先进的开源权重和闭源权重CUA开展了大规模评估,结果显示:仅通过纯视觉交互时,它们的表现远逊于专用编程智能体;但当同款CUA仅能直接访问文件编辑和bash操作这两个API时,性能大幅提升,尽管采用任务无关设计,其表现往往能达到专用智能体的水平;此外,若通过文本API为其提供更多IDE工具访问权限,所有模型的性能都会进一步提升。我们的分析表明,当前CUA的不足主要源于视觉 grounding 能力有限,且无法充分利用丰富的环境资源,这为未来改进指明了明确方向。PwP将软件工程确立为基准测试的天然领域,可用于检验通用计算机使用智能体能否在复杂任务上达到专家级性能。代码与数据已发布于https://programmingwithpixels.com

英文摘要:

Computer-use agents (CUAs) hold the promise of performing a wide variety of general tasks, but current evaluations have primarily focused on simple scenarios. It therefore remains unclear whether such generalist agents can automate more sophisticated and specialized work such as software engineering (SWE). To investigate this, we introduce $\texttt{Programming with Pixels}$ (PwP), the first comprehensive computer-use environment for software engineering, where agents visually control an IDE to perform diverse software engineering tasks. To enable holistic evaluation, we also introduce \texttt{PwP-Bench}, a benchmark of 15 existing and new software-engineering tasks spanning multiple modalities, programming languages, and skillsets. We perform an extensive evaluation of state-of-the-art open-weight and closed-weight CUAs and find that when interacting purely visually, they perform significantly worse than specialized coding agents. However, when the same CUAs are given direct access to just two APIs-file editing and bash operations-performance jumps, often reaching the levels of specialized agents despite having a task-agnostic design. Furthermore, when given access to additional IDE tools via text APIs, all models show further gains. Our analysis shows that current CUAs fall short mainly due to limited visual grounding and the inability to take full advantage of the rich environment, leaving clear room for future improvements.PwP establishes software engineering as a natural domain for benchmarking whether generalist computer-use agents can reach specialist-level performance on sophisticated tasks. Code and data released at https://programmingwithpixels.com

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑