TUA-Bench：通用终端使用代理的基准测试
TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents
AI总结：
提出TUA-Bench，包含120个跨五类任务的终端代理基准，涵盖日常数字活动与科学工程工作流，通过执行评分评估，发现最强模型Claude Opus 4.8仅达65.8%，揭示通用终端代理的显著差距。
摘要：
随着大型语言模型和框架的持续进步，在终端中运行的代理越来越能够执行除编码之外更广泛的通用计算机任务。然而，现有基准未能充分评估通用终端计算机使用代理（TUA）：通用计算机使用基准主要针对图形用户界面（GUI），而基于终端的基准则主要强调历史上原生在shell中的技术和编程中心化工作流。我们引入了TUA-Bench，一个用于终端使用代理的通用基准。TUA-Bench包含跨五个任务家族的120个真实世界任务，涵盖日常数字活动——包括文档编辑、电子邮件管理和实时网页信息检索——以及与博士级领域专家共同设计的需要专业软件的科学和工程工作流。这种广度使TUA-Bench区别于先前专注于shell或特定领域的基准。每个任务都经过手动设计，在具有确定性设置脚本的真实终端中运行，并通过基于执行的评分协议进行评估。我们发现，最强的前沿代理Claude Code with Claude Opus 4.8最大推理努力实现了65.8%的整体性能，在两个轨道上均存在显著差距。通过提供对终端使用能力的广泛且真实的评估，TUA-Bench旨在加速从狭窄的、特定任务的助手向能够在多样化数字环境中可靠运行的通用代理的转变。
英文摘要：
As large language models and harness frameworks continue to advance, agents operating in terminals are increasingly capable of performing a broader range of general computer-use tasks beyond coding. However, existing benchmarks do not adequately evaluate general-purpose terminal computer-use agents (TUAs): general computer-use benchmarks primarily target graphical user interfaces (GUIs), whereas terminal-based benchmarks largely emphasize technical and programming-centric workflows historically native to the shell. We introduce TUA-Bench, a general-purpose benchmark for terminal-use agents. TUA-Bench includes 120 real-world tasks across five task families, covering routine digital activities-including document editing, email management, and live-web information seeking-as well as scientific and engineering workflows co-designed with PhD-level domain experts that require specialized software. This breadth distinguishes TUA-Bench from prior shell-focused or domain-specific benchmarks. Each task is manually designed, runs in a real terminal with a deterministic setup script, and is evaluated by an execution-based scoring protocol. We find that the strongest frontier agent, Claude Code with Claude Opus 4.8 max reasoning effort, achieves 65.8% overall performance, with substantial gaps across both tracks. By providing a broad and realistic evaluation of terminal-use capabilities, TUA-Bench aims to accelerate the transition from narrow, task-specific assistants to general-purpose agents capable of operating reliably across diverse digital environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
