Trifuse: 通过多模态融合增强基于注意力的GUI定位
Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
AI总结：
Trifuse通过多模态融合提升GUI定位性能，整合注意力、OCR文本和图标描述语义，无需任务微调即可实现高效定位。
摘要：
GUI定位将自然语言指令映射到正确的界面元素，为GUI代理提供感知基础。现有方法主要依赖于通过大规模GUI数据集微调多模态大语言模型（MLLMs）来预测目标元素坐标，这需要大量数据且泛化能力差。近期基于注意力的替代方法利用MLLMs注意力机制中的定位信号，但因GUI图像中缺乏显式且互补的空间锚点而可靠性低。为解决这一限制，我们提出了Trifuse，一种基于注意力的定位框架，明确整合互补空间锚点。Trifuse通过Consensus-SinglePeak（CS）融合策略整合注意力、OCR衍生的文本提示和图标级别的描述语义，在保持定位尖峰的同时强制跨模态一致性。在四个定位基准测试中的广泛评估表明，Trifuse在无需任务特定微调的情况下实现了强劲性能，显著减少了对昂贵标注数据的依赖。此外，消融研究揭示，整合OCR和描述提示能跨不同主干网络一致提升基于注意力的定位性能，突显其作为GUI定位通用框架的有效性。
英文摘要：
GUI grounding maps natural language instructions to the correct interface elements, serving as the perception foundation for GUI agents. Existing approaches predominantly rely on fine-tuning multimodal large language models (MLLMs) using large-scale GUI datasets to predict target element coordinates, which is data-intensive and generalizes poorly to unseen interfaces. Recent attention-based alternatives exploit localization signals in MLLMs attention mechanisms without task-specific fine-tuning, but suffer from low reliability due to the lack of explicit and complementary spatial anchors in GUI images. To address this limitation, we propose Trifuse, an attention-based grounding framework that explicitly integrates complementary spatial anchors. Trifuse integrates attention, OCR-derived textual cues, and icon-level caption semantics via a Consensus-SinglePeak (CS) fusion strategy that enforces cross-modal agreement while retaining sharp localization peaks. Extensive evaluations on four grounding benchmarks demonstrate that Trifuse achieves strong performance without task-specific fine-tuning, substantially reducing the reliance on expensive annotated data. Moreover, ablation studies reveal that incorporating OCR and caption cues consistently improves attention-based grounding performance across different backbones, highlighting its effectiveness as a general framework for GUI grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
