arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23615cs.CV
收藏

无需标注的高分辨率大多模态模型视觉推理技术

HART: High-Resolution Annotation-Free Reasoning Technique through a Closed-loop Framework

State Key Laboratory of Novel Software Technology(新型软件技术国家重点实验室);Institute of Brain-inspired Intelligence(脑启发式智能研究院);Shenzhen Research Institute of Nanjing University(南京大学深圳研究院)

Jiacheng Yang, Anqi Chen, Yunkai Dang, Qi Fan, Cong Wang, Wenbin Li, Feng Miao, Yang Gao

AI总结:

HART通过闭环框架和AP-GRPO方法,实现无需外部标注的高分辨率视觉推理,提升多模态模型在高分辨率任务中的性能。

摘要:

当前的大多模态模型(LMMs)在处理高分辨率视觉输入时存在困难,因为图像标记的数量随着分辨率的增加而平方增长,导致大量冗余和无关信息。一种常见的做法是在推理过程中识别关键图像区域并参考其高分辨率对应物,通常通过外部视觉监督进行训练。然而,这种视觉监督线索需要从人类标注员处获取成本高昂的接地标签。同时,如何增强模型的接地能力以支持推理而不依赖额外标注仍是一个开放性问题。在本文中,我们提出了高分辨率无标注推理技术(HART),一种闭环框架,使LMMs能够聚焦并自我验证高分辨率视觉输入的关键区域。HART采用一种后训练范式,其中我们设计了优势偏好群体相对策略优化(AP-GRPO)以鼓励准确定位关键区域而无需外部视觉标注。值得注意的是,HART提供了可解释的推理路径,并实现了定位的高效优化。在MME-RealWorld-Lite、TreeBench、V* Bench、HR-Bench-4K/8K和MMStar上的大量实验表明,HART在广泛的高分辨率视觉任务中提升了性能,一致优于强大的基线模型。

英文摘要:

Current Large Multimodal Models (LMMs) struggle with high-resolution visual inputs during the reasoning process, as the number of image tokens increases quadratically with resolution, introducing substantial redundancy and irrelevant information. A common practice is to identify key image regions and refer to their high-resolution counterparts during reasoning, typically trained with external visual supervision. However, such visual supervision cues require costly grounding labels from human annotators. Meanwhile, it remains an open question how to enhance a model's grounding abilities to support reasoning without relying on additional annotations. In this paper, we propose High-resolution Annotation-free Reasoning Technique (HART), a closed-loop framework that enables LMMs to focus on and self-verify key regions of high-resolution visual inputs. HART incorporates a post-training paradigm in which we design Advantage Preference Group Relative Policy Optimization (AP-GRPO) to encourage accurate localization of key regions without external visual annotations. Notably, HART provides explainable reasoning pathways and enables efficient optimization of localization. Extensive experiments on MME-RealWorld-Lite, TreeBench, V* Bench, HR-Bench-4K/8K, and MMStar demonstrate that HART improves performance across a wide range of high-resolution visual tasks, consistently outperforming strong baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑