OS-SPEAR:一个用于操作系统代理安全、性能、效率和鲁棒性分析的工具包
OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents
AI总结:
OS-SPEAR通过四个维度系统分析操作系统代理,解决现有评估方法在安全、效率和多模态鲁棒性方面的不足,揭示代理在效率与安全之间的权衡及不同模态的鲁棒性差异。
摘要:
多模态大语言模型(MLLMs)的发展使关注点从文本生成转向主动行为执行,特别是通过操作系统代理在复杂GUI中导航。然而,这些代理转变为可信日常伙伴受到严格评估的阻碍,现有基准在安全场景狭窄、轨迹标记噪声和鲁棒性度量有限。为此,我们提出OS-SPEAR,一个用于系统分析操作系统代理的综合性工具包,涵盖安全、性能、效率和鲁棒性四个维度。OS-SPEAR引入四个专门子集:(1)安全子集涵盖多样化的环境和人类诱导的危险;(2)性能子集通过轨迹价值估计和分层抽样进行编纂;(3)效率子集通过时间延迟和token消耗双重视角量化性能;(4)鲁棒性子集通过跨模态干扰对视觉和文本输入进行测试。此外,我们提供自动化分析工具生成可读诊断报告。我们对22种流行的OS代理进行了广泛评估。实验结果揭示了当前现状的关键洞察:效率与安全或鲁棒性之间普遍存在权衡,专用代理在性能上优于通用模型,不同模态的鲁棒性差异显著。通过提供多维排名和标准化评估框架,OS-SPEAR为开发下一代可靠高效的OS代理提供了基础资源。数据集和代码可在https://github.com/Wuzheng02/OS-SPEAR获取。
英文摘要:
The evolution of Multimodal Large Language Models (MLLMs) has shifted the focus from text generation to active behavioral execution, particularly via OS agents navigating complex GUIs. However, the transition of these agents into trustworthy daily partners is hindered by a lack of rigorous evaluation regarding safety, efficiency, and multi-modal robustness. Current benchmarks suffer from narrow safety scenarios, noisy trajectory labeling, and limited robustness metrics. To bridge this gap, we propose OS-SPEAR, a comprehensive toolkit for the systematic analysis of OS agents across four dimensions: Safety, Performance, Efficiency, and Robustness. OS-SPEAR introduces four specialized subsets: (1) a S(afety)-subset encompassing diverse environment- and human-induced hazards; (2) a P(erformance)-subset curated via trajectory value estimation and stratified sampling; (3) an E(fficiency)-subset quantifying performance through the dual lenses of temporal latency and token consumption; and (4) a R(obustness)-subset that applies cross-modal disturbances to both visual and textual inputs. Additionally, we provide an automated analysis tool to generate human-readable diagnostic reports. We conduct an extensive evaluation of 22 popular OS agents using OS-SPEAR. Our empirical results reveal critical insights into the current landscape: notably, a prevalent trade-off between efficiency and safety or robustness, the performance superiority of specialized agents over general-purpose models, and varying robustness vulnerabilities across different modalities. By providing a multidimensional ranking and a standardized evaluation framework, OS-SPEAR offers a foundational resource for developing the next generation of reliable and efficient OS agents. The dataset and codes are available at https://github.com/Wuzheng02/OS-SPEAR.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。