FineState-Bench：面向细粒度GUI状态设置的状态条件化基准测试
FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting
AI总结：
FineState-Bench通过精确目标状态评估，检验智能体能否正确映射指令至UI控件，提出FineState-Metrics与VDA诊断工具，实验显示视觉接地仍有提升空间，但整体准确率不足。
摘要：
尽管大视觉-语言模型（LVLMs）快速发展，细粒度、状态条件化的GUI交互仍具挑战性。现有评估覆盖有限、目标状态定义不精确且过度依赖最终任务成功，掩盖了智能体失败的原因。为解决这一差距，我们引入FineState-Bench基准测试，评估智能体是否能正确将指令映射到目标UI控件并达到精确目标状态。FineState-Bench涵盖桌面、网页和移动平台共计2209个实例，覆盖四种交互家族和23种UI组件类型，每个实例均明确指定精确的目标状态。我们进一步提出FineState-Metrics，一个四阶段诊断流程，包含阶段性成功率：定位成功率（SR@Loc）、交互成功率（SR@Int）、定位精确状态成功率（ES-SR@Loc）和交互精确状态成功率（ES-SR@Int），以及可插拔的视觉诊断助手（VDA），生成描述和边界框定位提示以诊断视觉接地原因。在FineState-Bench上，精确目标状态成功率仍较低：ES-SR@Int在网页上最高达32.8%，平均跨平台为22.8%。使用VDA定位提示后，Gemini-2.5-Flash在ES-SR@Int上提升14.9个百分点，表明视觉接地改进有较大提升空间，但整体准确率仍不足以支持可靠的细粒度状态条件化交互。
英文摘要：
Despite the rapid progress of large vision-language models (LVLMs), fine-grained, state-conditioned GUI interaction remains challenging. Current evaluations offer limited coverage, imprecise target-state definitions, and an overreliance on final-task success, obscuring where and why agents fail. To address this gap, we introduce \textbf{FineState-Bench}, a benchmark that evaluates whether an agent can correctly ground an instruction to the intended UI control and reach the exact target state. FineState-Bench comprises 2,209 instances across desktop, web, and mobile platforms, spanning four interaction families and 23 UI component types, with each instance explicitly specifying an exact target state for fine-grained state setting. We further propose \textit{FineState-Metrics}, a four-stage diagnostic pipeline with stage-wise success rates: Localization Success Rate (SR@Loc), Interaction Success Rate (SR@Int), Exact State Success Rate at Locate (ES-SR@Loc), and Exact State Success Rate at Interact (ES-SR@Int), and a plug-and-play \textit{Visual Diagnostic Assistant} (VDA) that generates a Description and a bounding-box Localization Hint to diagnose visual grounding reason via controlled w/ vs.\ w/o comparisons. On FineState-Bench, exact goal-state success remains low: ES-SR@Int peaks at 32.8\% on Web and 22.8\% on average across platforms. With VDA localization hints, Gemini-2.5-Flash gains +14.9 ES-SR@Int points, suggesting substantial headroom from improved visual grounding, yet overall accuracy is still insufficient for reliable fine-grained state-conditioned interaction \href{https://github.com/FengxianJi/FineState-Bench}{Github.}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
