面向计算机使用代理的历史感知视觉基础批评家
A History-Aware Visually Grounded Critic for Computer Use Agents
AI总结：
提出HiViG框架，通过历史感知的视觉基础多模态批评家，在测试时评估动作并拦截错误，在多个GUI基准上提升成功率。
摘要：
针对计算机使用代理（CUA）的各种测试时干预措施（包括批评模型）已被开发出来，通过在复杂图形用户界面（GUI）环境中执行前动作评估来提高性能。然而，现有的批评家存在两个关键限制：（1）主要关注短视决策循环（例如，遗忘早期动作）；（2）缺乏检测有缺陷动作（例如，点击错误的UI元素）所需的视觉基础。为了解决这些问题，我们引入了HiViG，一个历史感知的视觉基础测试时框架，其核心是一个在真实GUI轨迹上训练的多模态批评家，用于将过去的交互抽象为紧凑记录，并基于视觉基础评估动作。在测试时，HiViG将批评家集成到策略决策循环中，以提供宏观动作历史（总结策略已完成成就）和视觉基础批评（根据当前截图验证原始执行坐标，在执行前拦截错误）。在网页、移动和桌面基准测试中，HiViG持续优于现有的标量和口头批评家，在Qwen3-VL-32B上比最强基线平均成功率提高5.8%，在Gemini-3-Flash上提高9.0%，并展示了强大的跨平台泛化能力。消融实验表明，宏观动作历史缓解了短视规划，视觉基础批评减少了执行错误，这两个组件对于长时域GUI任务中的测试时扩展至关重要。
英文摘要：
Various test-time interventions for Computer Use Agents (CUAs), including critic models, have been developed to improve performance through pre-execution action evaluation in complex Graphical User Interface (GUI) environments. However, existing critics suffer from two key limitations: they (1) focus primarily on short-sighted decision loops (e.g., forgetting earlier actions) and (2) lack the visual grounding needed to detect flawed actions (e.g., clicking wrong UI elements). To address these, we introduce HiViG, a History-aware Visually Grounded test-time framework, built around a multimodal critic trained on real GUI trajectories to abstract past interactions into a compact record and to evaluate actions with visual grounding. At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history, which summarizes the policy's completed achievements, and visually grounded critique, which verifies raw execution coordinates against the current screenshot to intercept errors before execution. Across web, mobile, and desktop benchmarks, HiViG consistently outperforms existing scalar and verbal critics, improving average success rates over the strongest baseline by 5.8% for Qwen3-VL-32B and 9.0% for Gemini-3-Flash, and demonstrates strong cross-platform generalization. Ablations show that macro-action history mitigates short-sighted planning and visually grounded critique reduces execution errors, with both components being critical for test-time scaling in long-horizon GUI tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
