arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.00935cs.CRcs.AI
收藏

评估使用计算机的智能体的危害性

Measuring Harmfulness of Computer-Using Agents

Arizona State University(亚利桑那州立大学);University of California, Berkeley(加州大学伯克利分校)

Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang, Ji Wang, Tianyu Shi, Jiaxin Wen

AI总结:

本研究推出CUAHarm基准评估使用计算机的智能体的滥用风险,发现前沿大模型作为该类智能体时滥用风险高,监控其操作难度大,分层摘要策略可小幅提升监控效果。

摘要:

能够自主控制计算机执行多步骤操作的使用计算机的智能体(Computer-Using Agents, CUAs)若被滥用,可能会带来重大安全风险。然而,现有基准主要评估大型语言模型(Large Language Models, LMs)在聊天机器人或简单工具使用场景中的表现。为更全面地评估CUAs的滥用风险,我们推出了新基准CUAHarm,其包含104项由专家编写的现实滥用风险场景,例如禁用防火墙、泄露数据或安装后门。我们提供带有基于规则的可验证奖励的沙箱,以衡量CUAs执行这些任务的成功率(例如防火墙是否确实被禁用),而非仅评估拒绝率。我们评估了前沿大型语言模型,包括GPT-5、Claude 4 Sonnet、Gemini 2.5 Pro、Llama-3.3-70B和Mistral Large 2。即使没有越狱提示,这些前沿大型语言模型执行这些恶意任务的成功率也很高,例如Gemini 2.5 Pro的成功率达90%。此外,尽管较新的模型在以往安全基准中更安全,但它们作为CUAs的滥用风险却更高,例如Gemini 2.5 Pro比Gemini 1.5 Pro风险更高。另外,当这些大型语言模型作为聊天机器人时,对常见恶意提示(例如制造炸弹)具有鲁棒性,但作为CUAs时仍可能表现不安全。我们进一步评估了领先的智能体框架UI-TARS-1.5,发现其虽提升了性能,但也放大了滥用风险。为缓解CUAs的滥用风险,我们探索使用大型语言模型监控CUAs的操作,发现监控不安全的计算机使用操作比监控传统不安全的聊天机器人响应困难得多。尽管监控思维链(chain-of-thought)能带来适度提升,但平均监控准确率仅为77%。分层摘要策略可将性能提升最多13%,尽管监控仍不可靠,但这是一个有前景的研究方向。CUAHarm已在该httpsURL发布,以推动进一步研究。

英文摘要:

Computer-using agents (CUAs), which can autonomously control computers to perform multi-step actions, might pose significant safety risks if misused. However, existing benchmarks mainly evaluate LMs in chatbots or simple tool use. To more comprehensively evaluate CUAs' misuse risks, we introduce a new benchmark: CUAHarm. CUAHarm consists of 104 expert-written realistic misuse risks, such as disabling firewalls, leaking data, or installing backdoors. We provide a sandbox with rule-based verifiable rewards to measure CUAs' success rates in executing these tasks (e.g., whether the firewall is indeed disabled), beyond refusal rates. We evaluate frontier LMs including GPT-5, Claude 4 Sonnet, Gemini 2.5 Pro, Llama-3.3-70B, and Mistral Large 2. Even without jailbreaking prompts, these frontier LMs comply with executing these malicious tasks at a high success rate (e.g., 90% for Gemini 2.5 Pro). Furthermore, while newer models are safer in previous safety benchmarks, their misuse risks as CUAs become even higher, e.g., Gemini 2.5 Pro is riskier than Gemini 1.5 Pro. Additionally, while these LMs are robust to common malicious prompts (e.g., creating a bomb) when acting as chatbots, they could still act unsafely as CUAs. We further evaluate a leading agentic framework (UI-TARS-1.5) and find that while it improves performance, it also amplifies misuse risks. To mitigate the misuse risks of CUAs, we explore using LMs to monitor CUAs' actions. We find monitoring unsafe computer-using actions is significantly harder than monitoring conventional unsafe chatbot responses. While monitoring chain-of-thoughts leads to modest gains, the average monitoring accuracy is only 77%. A hierarchical summarization strategy improves performance by up to 13%, a promising direction though monitoring remains unreliable. CUAHarm is released at https://github.com/db-ol/CUAHarm to facilitate further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑