PrecisionCUA：代码编辑器中像素级光标定位的迭代视觉细化
PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors
AI总结：
提出PrecisionCUA方法，通过迭代视觉反馈细化机制实现代码编辑器中像素级光标定位，显著提升点击精度和任务成功率。
摘要：
计算机使用代理（CUA）从根本上依赖图形用户界面（GUI）定位，将语言指令转化为可执行的屏幕操作，但在密集编码界面（如VS Code和Cursor）中，需要亚像素精度才能与密集IDE元素交互的编辑级定位尚未得到充分探索。现有方法通常依赖单次坐标预测，缺乏纠错机制，在高密度界面中常常失败。在本技术报告中，我们对编码环境中的像素级光标定位进行了实证研究。我们的代理不是单步执行，而是参与迭代细化过程，利用先前尝试的视觉反馈来达到目标元素。这种闭环定位机制使代理能够自我纠正位移误差并适应动态UI变化。我们在Claude、Qwen和GPT上的一系列复杂编码基准上评估了我们的方法，结果表明多轮细化在点击精度和整体任务成功率上均显著优于最先进的单次模型。我们的结果表明，迭代视觉推理是下一代可靠软件工程代理的关键组成部分。代码：https://github.com/microsoft/precision-cua-bench/tree/main。
英文摘要：
Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions, but editing-level grounding in dense coding interfaces (such as VS Code and Cursor), where sub-pixel accuracy is required to interact with dense IDE elements, remains underexplored. Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces. In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments. Instead of a single-step execution, our agent engages in an iterative refinement process, utilizing visual feedback from previous attempts to reach the target element. This closed-loop grounding mechanism allows the agent to self-correct displacement errors and adapt to dynamic UI changes. We evaluate our approach across Claude, Qwen, and GPT on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and overall task success rate. Our results suggest that iterative visual reasoning is a critical component for the next generation of reliable software engineering agents. Code: https://github.com/microsoft/precision-cua-bench/tree/main.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
