发表机构
Baidu, Inc.; Ningxia Electric Power Engineering Co., Ltd.（百度公司; 宁夏电力工程有限公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
OmegaUse-SOP是一种人在环SOP工程系统，通过四个模块将专业计算机使用的人类演示转化为GUI智能体可复用技能，在PVsyst 7.2光伏模拟测试中提升了GUI智能体在专业SOP任务上的可靠性。
AI 中文摘要
大型语言模型（LLMs）正逐渐从对话助手演变为能够操作外部数字环境的智能体。图形用户界面（GUI）智能体在这一转变中发挥着重要作用，因为许多现实世界的工作流程仍仅能通过面向用户的软件界面访问。然而，尽管通用计算机使用基准测试已取得进展，特定领域的专业标准操作程序（SOP）对GUI智能体而言仍具挑战性，因为它们通常涉及隐性领域知识、软件特定约定以及任务级验证要求。我们引入OmegaUse-SOP，这是一种人在环的SOP工程系统，用于将专业计算机使用的人类演示转化为GUI智能体可复用的SOP技能。类似于提示工程，SOP工程迭代优化演示、执行规则和领域知识，将专业SOP转化为可复用的GUI智能体技能。OmegaUse-SOP由四个模块组成：观察（Observe）、推理（Reason）、配置（Configure）和执行（Execute）。这些模块协同工作，将专家操作记录为多模态GUI轨迹，将低级事件抽象为语义步骤级指令，整合领域规则和任务特定参数，并通过逐步接地、动作生成和验证在实时GUI环境中执行生成的技能。为验证其有效性，我们与一家电力行业客户合作，在PVsyst 7.2中的光伏模拟工作流程上测试OmegaUse-SOP。结果表明，OmegaUse-SOP可提高GUI智能体在专业SOP任务上的可靠性，为在特定领域专业软件环境中部署GUI智能体提供了可行路径。
英文摘要
Large language models (LLMs) are increasingly evolving from conversational assistants into agents capable of operating external digital environments. Graphical user interface (GUI) agents play an important role in this transition, as many real-world workflows remain accessible only through user-facing software interfaces. However, despite recent progress on general computer-use benchmarks, domain-specific professional standard operating procedures (SOPs) remain challenging for GUI agents because they often involve implicit domain knowledge, software-specific conventions, and task-level verification requirements. We introduce OmegaUse-SOP, a human-in-the-loop SOP Engineering system for transforming human demonstrations of professional computer use into reusable SOP skills for GUI agents. Analogous to prompt engineering, SOP Engineering iteratively refines demonstrations, execution rules, and domain knowledge to convert professional SOPs into reusable GUI-agent skills. OmegaUse-SOP consists of four modules: Observe, Reason, Configure, and Execute. Together, these modules record expert operations as multimodal GUI traces, abstract low-level events into semantic step-level instructions, incorporate domain rules and task-specific parameters, and execute the resulting skills in live GUI environments through step-wise grounding, action generation, and verification. To demonstrate its effectiveness, we collaborate with a power-sector client and test OmegaUse-SOP on photovoltaic simulation workflows in PVsyst 7.2. The results suggest that OmegaUse-SOP can improve GUI-agent reliability on professional SOP tasks, highlighting a practical path toward deploying GUI agents in domain-specific professional software environments.
CommentsAccpeted to EMNLP 2026 demo track
DOI：10.21203/rs.3.rs-10741604/v1
