LongCLI-Bench: 一个初步的基准测试与研究,用于命令行界面中的长周期代理编程
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
AI总结:
LongCLI-Bench通过长周期任务评估代理能力,揭示现有代理在复杂任务中的不足,强调人机协作的重要性。
摘要:
近年来,人工智能辅助编程的进步使代理能够通过命令行界面执行复杂的工作流,然而,现有基准测试受限于短任务周期、来自GitHub抓取的数据污染以及缺乏细粒度的评估指标,无法严格评估现实软件工程中至关重要的长周期规划和执行能力。为解决这些差距,我们引入了LongCLI-Bench,一个全面的基准测试,旨在评估长周期、现实任务中的代理能力。我们从超过1000个计算机科学作业和现实工作流中精选了20个高质量、长周期任务,涵盖四个工程类别:从零开始、功能添加、故障修复和重构。我们提出了一种双集测试协议用于LongCLI-Bench,该协议测量需求满足(失败-通过)和回归避免(通过-通过),并结合步骤级评分以定位执行失败。广泛的实验表明,即使是最先进的代理在LongCLI-Bench中的通过率也低于20%。步骤级分析进一步表明,大多数任务在完成度低于30%时停滞,突显关键失败往往发生在早期阶段。尽管自我纠正带来微小提升,但通过计划注入和交互式指导的人机协作则带来显著提升。这些结果强调,未来研究必须强调协同人机工作流的发展,同时推进代理规划和执行能力的提升,以克服长周期任务性能中的关键挑战。
英文摘要:
Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。