面向高效自我中心 grounding 的动态分辨率路由

中文摘要 AI 辅助

自我中心视觉 grounding 需要高分辨率输入来定位小物体，但将多模态大语言模型扩展到该领域受到视觉 token 处理过高成本的限制。我们发现当前基于 token 缩减的高效策略不可靠，无法选择以物体为中心的空间证据。为解决这一问题，我们提出 SmartRes，这是一个通过动态分辨率路由在像素空间中执行效率优化的框架。SmartRes 首先对低分辨率视图进行编码以获取全局上下文，并使用轻量级路由器激活以物体为中心区域的高分辨率 patch，构建保序视觉序列。为进一步在严重的前景-背景不平衡下实现鲁棒路由，我们引入了边际正则化路由目标，该目标可增加前景-背景 logit 分离度并提高前景召回率。在 Ego4D 和 EgoIntention 上的实验表明，SmartRes 最多可减少 67% 的视觉 token，同时保留全分辨率性能的 86.4%，与最先进的 token 缩减方法相比，推理速度最多可提升 1.66 倍且精度更高。此外，在小物体 grounding 上的出色性能表明 SmartRes 对自我中心应用的有效性，代码将公开提供。

英文摘要

Egocentric visual grounding requires high-resolution inputs to localize small objects. However, scaling Multimodal Large Language Models to this domain is constrained by the excessive cost of visual token processing. We identify that current efficient strategies based on token reduction are unreliable for selecting object-centric spatial evidence. To overcome this, we propose SmartRes, a framework that performs efficiency optimization in the pixel space via dynamic resolution routing. SmartRes first encodes a low-resolution view for global context and uses a lightweight router to activate high-resolution patches in object-centric regions and constructs an order-preserving visual sequence. To further enable robust routing under severe foreground-background imbalance, we introduce a margin-regularized routing objective that increases foreground-background logit separation and improves foreground recall. Experiments on Ego4D and EgoIntention show that SmartRes reduces visual tokens by up to 67% while retaining 86.4% of full-resolution performance, and achieves up to 1.66X faster inference than state-of-the-art token reduction methods with higher accuracy. Furthermore, strong performance on small object grounding indicates the effectiveness of SmartRes towards egocentric applications. Code will be publicly available.

发表机构

• National University of Singapore（新加坡国立大学）
• The Hong Kong University of Science and Technology（香港科技大学）
• Nanyang Technological University（南洋理工大学）

机构由 AI 辅助整理，请以论文原文为准。