ScreenParse：超越稀疏标注的完整屏幕解析监督
ScreenParse: Moving Beyond Sparse Grounding with Complete Screen Parsing Supervision
AI总结：
ScreenParse通过大规模完整屏幕解析标注，训练出性能优异的ScreenVLM模型，显著提升了密集解析能力和迁移表现。
摘要：
现代计算机使用代理（CUA）必须将屏幕视为结构化状态，在执行任务前需感知可见元素的位置、文本内容等。然而，现有标注数据集提供稀疏监督，限制了覆盖范围和泛化能力。本文提出ScreenParse数据集，包含771K网页截图的密集标注，通过自动化流程生成。基于该数据集训练的ScreenVLM模型在密集解析任务中表现优异，并在迁移学习中展现出强大的结构先验能力。
英文摘要：
Modern computer-use agents (CUA) must perceive a screen as a structured state, what elements are visible, where they are, and what text they contain, before they can reliably ground instructions and act. Yet, most available grounding datasets provide sparse supervision, with insufficient and low-diversity labels that annotate only a small subset of task-relevant elements per screen, which limits both coverage and generalization; moreover, practical deployment requires efficiency to enable low-latency, on-device use. We introduce ScreenParse, a large-scale dataset for complete screen parsing, with dense annotations of all visible UI elements (boxes, 55-class types, and text) across 771K web screenshots (21M elements). ScreenParse is generated by Webshot, an automated, scalable pipeline that renders diverse urls, extracts annotations and applies VLM-based relabeling and quality filtering. Using ScreenParse, we train ScreenVLM, a compact, 316M-parameter vision language model (VLM) that decodes a compact ScreenTag markup representation with a structure-aware loss that upweights structure-critical tokens. ScreenVLM substantially outperforms much larger foundation VLMs on dense parsing (e.g., 0.592 vs. 0.294 PageIoU on ScreenParse) and shows strong transfer to public benchmarks. Moreover, finetuning foundation VLMs on ScreenParse consistently improves their grounding performance, suggesting that dense screen supervision provides transferable structural priors for UI understanding. Project page: https://saidgurbuz.github.io/screenparse/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
