arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02419cs.AIcs.SE
收藏

SafeGround: 通过不确定性校准了解何时信任GUI接地模型

SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration

University of California, Santa Barbara（加州大学圣芭芭拉分校）；University of California, Santa Cruz（加州大学圣克ruz分校）

Qingni Wang, Yue Fan, Xin Eric Wang

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

SafeGround通过不确定性校准提升GUI接地模型的可靠性与准确性。

摘要：

图形用户界面（GUI）接地旨在将自然语言指令转换为可执行的屏幕坐标，从而实现自动化的GUI交互。然而，错误的接地可能导致昂贵且难以逆转的操作（例如错误的付款批准），这引发了关于模型可靠性的担忧。在本文中，我们介绍了SafeGround，这是一种不确定性意识的框架，用于GUI接地模型，通过在测试前的校准来实现风险意识的预测。SafeGround利用一种分布感知的不确定性量化方法，以捕捉来自任何给定模型输出的随机样本的空间分散性。然后，通过校准过程，SafeGround推导出一个测试时的决策阈值，具有统计保证的假发现率（FDR）控制。我们将在多个GUI接地模型上应用SafeGround，以应对具有挑战性的ScreenSpot-Pro基准测试。实验结果表明，我们的不确定性度量在区分正确和错误预测方面始终优于现有基线，而校准的阈值能够可靠地实现严格的风险控制和潜在的系统级准确性提升。在多个GUI接地模型上，SafeGround在Gemini-only推理上将系统级准确性提高了最高5.38个百分点。

英文摘要：

Graphical User Interface (GUI) grounding aims to translate natural language instructions into executable screen coordinates, enabling automated GUI interaction. Nevertheless, incorrect grounding can result in costly, hard-to-reverse actions (e.g., erroneous payment approvals), raising concerns about model reliability. In this paper, we introduce SafeGround, an uncertainty-aware framework for GUI grounding models that enables risk-aware predictions through calibrations before testing. SafeGround leverages a distribution-aware uncertainty quantification method to capture the spatial dispersion of stochastic samples from outputs of any given model. Then, through the calibration process, SafeGround derives a test-time decision threshold with statistically guaranteed false discovery rate (FDR) control. We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark. Experimental results show that our uncertainty measure consistently outperforms existing baselines in distinguishing correct from incorrect predictions, while the calibrated threshold reliably enables rigorous risk control and potentials of substantial system-level accuracy improvements. Across multiple GUI grounding models, SafeGround improves system-level accuracy by up to 5.38% percentage points over Gemini-only inference.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑