VGent: 通过模块化设计实现视觉 grounding 的解耦推理与预测
VGent: Visual Grounding via Modular Design for Disentangling Reasoning and Prediction
AI总结：
VGent 通过模块化设计实现视觉 grounding 的解耦推理与预测，利用冻结 MLLM 和解码器交叉注意力机制，提升多目标识别性能。
摘要：
当前的视觉 grounding 模型要么基于多模态大语言模型（MLLM），该模型进行自回归解码，速度慢且有幻觉风险，要么通过重新对齐大语言模型与视觉特征来学习新的特殊或物体标记以实现 grounding，这可能会削弱大语言模型的预训练推理能力。相反，我们提出了 VGent，一种模块化编码器-解码器架构，明确解耦了高层推理和低层边界框预测。具体来说，一个冻结的 MLLM 作为编码器，提供未受干扰的强大推理能力，而解码器则将由检测器提出的高质量框作为查询，并通过在编码器的隐藏状态上进行交叉注意力来选择目标框（s）。这种设计充分利用了目标检测和 MLLM 的进展，避免了自回归解码的缺点，并实现了快速推理。此外，它支持对编码器和解码器的模块化升级以造福整个系统：我们引入（i）QuadThinker，一种基于强化学习的训练范式，用于增强编码器的多目标推理能力；（ii）mask-aware 标签用于解决检测-分割的歧义；以及（iii）全局目标识别，以提高所有目标的识别能力，这有助于在增强的提案中进行选择。在多目标视觉 grounding 基准测试上，实验表明 VGent 实现了新的 state-of-the-art，比先前方法在 F1 上提高了 +20.6%，并在视觉参考挑战中进一步提高了 gIoU 8.2% 和 cIoU 5.8%，同时保持了恒定且快速的推理延迟。
英文摘要：
Current visual grounding models are either based on a Multimodal Large Language Model (MLLM) that performs auto-regressive decoding, which is slow and risks hallucinations, or on re-aligning an LLM with vision features to learn new special or object tokens for grounding, which may undermine the LLM's pretrained reasoning ability. In contrast, we propose VGent, a modular encoder-decoder architecture that explicitly disentangles high-level reasoning and low-level bounding box prediction. Specifically, a frozen MLLM serves as the encoder to provide untouched powerful reasoning capabilities, while a decoder takes high-quality boxes proposed by detectors as queries and selects target box(es) via cross-attending on encoder's hidden states. This design fully leverages advances in both object detection and MLLM, avoids the pitfalls of auto-regressive decoding, and enables fast inference. Moreover, it supports modular upgrades of both the encoder and decoder to benefit the whole system: we introduce (i) QuadThinker, an RL-based training paradigm for enhancing multi-target reasoning ability of the encoder; (ii) mask-aware label for resolving detection-segmentation ambiguity; and (iii) global target recognition to improve the recognition of all the targets which benefits the selection among augmented proposals. Experiments on multi-target visual grounding benchmarks show that VGent achieves a new state-of-the-art with +20.6% F1 improvement over prior methods, and further boosts gIoU by +8.2% and cIoU by +5.8% under visual reference challenges, while maintaining constant, fast inference latency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
