驯服斯基拉：理解编码海洋中的多头代理守护程序
Taming Scylla: Understanding the multi-headed agentic daemon of the coding seas
AI总结：
本文提出Scylla框架，用于评估代理编码工具，通过结构化消融研究量化复杂性和效率的权衡，表明架构复杂性不总是提升质量。
摘要：
基于大语言模型的工具正在以快速的速度自动化更多的软件开发任务，但尚没有严格的方法来评估不同的架构选择——提示、技能、工具、多代理设置——如何实质性地影响能力和成本。本文介绍了Scylla，一个通过结构化消融研究进行代理编码工具基准测试的评估框架，使用七个测试层级（T0-T6）逐步增加复杂性以隔离直接影响结果的因素以及如何影响结果。关键指标是通过量化的成本（CoP）：获得一个正确解决方案的预期美元成本，直接量化复杂性和效率之间的权衡。该框架模型无关，设计用于任何CLI工具；本文使用Claude Sonnet 4.5演示，使用多个LLM评委（Opus 4.5、Sonnet 4.5、Haiku 4.5）从同一家供应商进行评估共识，其中评委使用直接测试、人类设计的LLM评估评分表和定性评估来评分。结果是一个可重复的框架，量化了代理复杂性和实际结果之间的权衡，表明架构复杂性并不总是提高质量。
英文摘要：
LLM-based tools are automating more software development tasks at a rapid pace, but there is no rigorous way to evaluate how different architectural choices -- prompts, skills, tools, multi-agent setups -- materially affect both capability and cost. This paper introduces Scylla, an evaluation framework for benchmarking agentic coding tools through structured ablation studies that uses seven testing tiers (T0-T6) progressively adding complexity to isolate what directly influences results and how. The key metric is Cost-of-Pass (CoP): the expected dollar cost to get one correct solution, which directly quantifies the trade-off between complexity and efficiency. The framework is model-agnostic, designed to work with any CLI tool; this paper demonstrates it with Claude Sonnet 4.5, using multiple LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) from the same vendor for evaluation consensus, where judges score results using direct tests, human-designed LLM-evaluated rubrics, and qualitative assessment. The result is a reproducible framework that quantifies trade-offs between agent complexity and actual outcomes, suggesting that architectural complexity does not always improve quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
