AgentMeter: 评估基于CLI的本地任务求解智能体的模型-CLI匹配
Matching Matters: A Fair Quality-Efficiency Benchmark for Command-Line Agents
AI总结:
提出AgentMeter基准,通过成功锚定、成本感知的AMS指标评估模型与CLI的匹配,发现模型和CLI选择不可解耦,需作为部署单元评估。
摘要:
LLM智能体越来越多地通过命令行和基于CLI的接口解决本地任务,包括代码编辑、仓库检查、数据分析和文件工作流。现有评估通常强调任务成功,但部署的本地智能体不仅仅是模型:CLI中介提示、上下文重放、工具输出、文件访问、终端观察和停止行为。因此,同一模型在不同CLI下可能产生不同的成功、令牌和成本曲线。我们引入AGENTMETER,一个用于评估CLI中介的本地任务求解智能体中模型-CLI匹配的基准,以及AgentMeter Score (AMS),一个在标定任务难度层级上的成功锚定、成本感知的度量。AgentMeter使用Benchmark90作为完整验证集,Core30作为低成本子集,用于跨24个完整模型-CLI配置的扩展比较。在Core30上,常见部署标准选择不同的配置:最高Pass/30选择GLM-5.1与qwen-coder,最低Tok./Pass选择GPT-5.3-Codex与kimi-cli,最低可计费USD/Pass选择Qwen3.6+与Codex,而最高AMS选择Qwen3.6+与kimi-cli。Benchmark90验证保留了Top-1配置和Top-3集合,Spearman相关系数0.765,Kendall相关系数0.567,AMS平均绝对误差0.0383。这些结果表明,模型选择和CLI选择不应解耦,模型-CLI配置应作为部署单元进行评估。
英文摘要:
Rapid advances in large language models have improved the task-solving capabilities of command-line-interface (CLI)-based agents, whose CLIs determine how models invoke tools, maintain interaction history, and recover from failures. Consequently, effective matching between CLIs and LLMs has become essential. However, existing agent benchmarks largely emphasize success rate while overlooking practical objectives such as cost and efficiency, as well as the selection of LM-CLI combinations, all of which are critical in real-world deployment. We therefore introduce AgentMeter, a quality-efficiency benchmark with a new metric, the AgentMeter Score (AMS), that jointly characterizes task quality, budget sensitivity, and resource-intensive zero-reward execution, enabling a more complete assessment of deployed LM-CLI pairs. Furthermore, collected task descriptions may inadvertently favor LM-CLI pairs that are particularly compatible with their wording and structure, causing evaluation results to reflect description-specific advantages rather than general task-solving capability. We therefore propose AgentMeter-Opt, a trajectory-grounded optimization framework that constructs pair-adapted, task-preserving description variants to build a fairer evaluation set across LM-CLI pairs. Extensive experiments show that no CLI is universally optimal across language models and that task success, execution cost, and AMS identify different competitive configurations. Results on AgentMeter-Opt further reveal that task-preserving description changes affect LM-CLI pairs unevenly and can alter their relative ordering across valid description conditions. Together, AgentMeter and AgentMeter-Opt provide a practical foundation for fair and deployment-relevant evaluation of command-line agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。