AI 中文总结
本研究提出MissClick攻击方法，针对GUI定位模型设计两种目标的白盒对抗攻击，在OS-Atlas和UGround数据集上大幅提升无目标与有目标攻击成功率。
AI 中文摘要
近期的GUI视觉定位模型会将屏幕坐标生成为数字令牌序列，这些序列会被解析为数值并映射为可执行的点击操作。这一坐标生成过程的安全隐患在很大程度上被忽视了。我们注意到，每个坐标数字都被预测为分类令牌，但在解析后，百位数字每变化1，对应的坐标分量就会变化100个单位，这会导致执行的点击产生大幅位移。这一发现促使我们提出攻击目标，该目标需考虑坐标输出的数值和位值结构，而非将其视为普通文本。此外，无目标攻击和有目标攻击有不同的成功条件——前者是将点击移出正确区域，后者是将点击移至攻击者指定的区域——因此适合采用不同的攻击目标。我们提出MissClick，这是一种简单有效的白盒对抗攻击，具有两个针对特定目标的目标：MissClick-U最大化软坐标位移以实现无目标干扰，MissClick-T最小化位加权目标数字损失以实现有目标劫持。与现有针对GUI定位模型的攻击在OS-Atlas和UGround上，覆盖桌面、网页和移动平台的测试相比，MissClick-U的无目标攻击成功率分别达到75.07%和72.93%（提升16.62和30.72个百分点），MissClick-T的有目标攻击成功率分别达到44.86%和62.67%（提升31.73和47.06个百分点）。攻击目标对比进一步显示，软坐标位移能实现最高的无目标攻击成功率，而位加权目标数字优化能实现最高的有目标攻击成功率，揭示了两种攻击目标具有不同的目标偏好。
英文摘要
Recent GUI visual grounding models generate screen coordinates as digit-token sequences that are parsed into numerical values and mapped to executable clicks. This generation-to-execution interface creates an attack surface that existing objectives over visual representations or coordinate-token sequences do not explicitly model. Although each coordinate digit is predicted as a token, its spatial effect after parsing depends on decimal position: changing a hundreds-place digit by one shifts the coordinate by 100 units, whereas the same change at the ones place shifts it by one. This mismatch motivates attack objectives that account for both numerical coordinate structure and click execution. Moreover, untargeted and targeted attacks require different objectives because they aim to move the click outside the correct region and into an attacker-specified region, respectively. We propose MissClick, an execution-aware white-box attack that aligns optimization with click-level success conditions. MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted redirection. On OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07% and 72.93% (+16.62 and +30.72 pp), while MissClick-T achieves targeted success rates of 44.86% and 62.67% (+31.73 and +47.06 pp). Among the evaluated objectives, soft-coordinate displacement performs best for untargeted attacks, whereas place-weighted target-digit optimization performs best for targeted attacks, supporting goal-specific execution-aware objective design.
