LUMOS：面向无障碍AI代理的语义操作系统层
LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents
AI总结：
提出LUMOS语义层，将操作系统无障碍元数据转化为机器可读蓝图，使AI代理通过结构化UI原语而非截图进行观察-行动循环，降低视觉依赖和成本。
摘要：
当前操作系统暴露的接口针对人类用户优化，而非AI代理。人类受益于像素、图标、窗口、视觉分组、鼠标移动和键盘快捷键；AI代理反而需要紧凑的语义状态、基础操作和可靠反馈。因此，许多计算机使用代理被迫解释截图、OCR输出和视觉裁剪，引入高令牌成本、视觉模糊、延迟和坐标不确定性。本文介绍LUMOS（语言模型统一机器可读操作系统语义），一个AI代理与操作系统之间的语义交互层。LUMOS将原生无障碍元数据和浏览器UI结构转换为具有稳定标识符、角色、名称、值、边界和操作能力的机器可读语义蓝图。它还通过操作系统自动化API查询光标下或附近的UI元素，支持实时语义指针定位。然后，LLM通过基于无障碍的观察-行动循环行动，使用受限的可见UI原语而非特定应用脚本。LUMOS并不声称取代视觉代理；相反，它减少了对截图的依赖，因为操作系统已经提供了语义结构。这些结果指向了AI原生操作系统和机器可读交互层的路径。
英文摘要：
Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
