ProcCtrlBench: 评估LLM编码智能体中的过程级缺陷与控制保持
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
AI总结：
提出ProcCtrlBench基准，通过可复用的缺陷本体和标准化轨迹表示，从过程证据而非仅最终结果评估LLM编码智能体的执行质量，并引入控制保持量化执行过程的可解释性、可中断性等属性。
摘要：
现有的LLM编码智能体基准主要评估最终结果。虽然有助于衡量整体能力，但这些指标提供的可见性有限，常常遗漏执行过程中出现的缺陷。我们提出了ProcCtrlBench，一个用于LLM编码智能体执行过程评估的基准。ProcCtrlBench将重复出现的执行缺陷组织成一个可复用的本体，涵盖4类11种缺陷类型，并通过标准化的过程证据而非仅最终结果来评估智能体轨迹。为了支持异构智能体之间的比较，ProcCtrlBench将原始日志标准化为统一的轨迹表示，并报告基于过程发现的校准评分卡。此外，ProcCtrlBench使用控制保持作为量化执行过程质量的方式，捕获执行是否保持可解释、可中断、可纠正、可逆，并在需要时能够交还控制权。我们在从三个基准（AndroidBench、TerminalBench和SWE-bench-Verified）中采样的200个案例上评估了ProcCtrlBench。结果表明，ProcCtrlBench可以以有用的可靠性实例化，提供比直接阈值化更稳定的语义，并揭示了传统基于结果的评估常常忽略的执行质量的有意义差异。
英文摘要：
Existing benchmarks for LLM coding agents primarily evaluate final outcomes. While useful for measuring overall capability, these metrics provide limited visibility and often miss defects that arise during execution. We present ProcCtrlBench, a benchmark for execution-process evaluation in LLM coding agents. ProcCtrlBench organizes recurrent execution defects into a reusable ontology covering 11 defect types in 4 categories, and evaluates agent trajectories through standardized process evidence rather than final outcomes alone. To support comparison across heterogeneous agents, ProcCtrlBench standardizes raw logs into a unified trajectory representation and reports calibrated scorecards over process-level findings. In addition, ProcCtrlBench uses control preservation as a way to quantify execution-process quality, capturing whether execution remains interpretable, interruptible, correctable, reversible, and able to hand back authority when needed. We evaluate ProcCtrlBench on 200 cases sampled from three benchmarks: AndroidBench, TerminalBench, and SWE-bench-Verified. Results show that ProcCtrlBench can be instantiated with useful reliability, provides more stable semantics than direct thresholding, and reveals meaningful differences in execution quality that are often overlooked by conventional outcome-based evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
