增强任意开源多模态大语言模型的部件级点定位能力
Enhancing Part-Level Point Grounding for Any Open-Source MLLMs
AI总结：
提出一种通用方法，通过冻结原模型参数并引入Q-Synth模块和注意力到点解码器，为任意开源MLLM赋予精确的2D部件级点定位能力，显著提升部件级定位精度。
摘要：
视觉定位旨在将自由形式的文本查询与图像中的特定区域关联起来。虽然最近的多模态大语言模型（MLLMs）在该领域展示了有前景的能力，但它们主要擅长对象级定位，而在部件级定位上往往表现不佳——这是机器人操作等细粒度任务的基本要求。在这项工作中，我们引入了一种通用方法，使任何开源MLLM具备精确的2D部件级点定位能力，提供了比传统定位表示更直接的替代方案。我们的方法利用了MLLM中固有的注意力机制。通过提出的Q-Synth模块在中间层合成文本条件、定位感知的查询，我们捕获目标相关的注意力模式，并使用轻量级的注意力到点解码器对其进行细化，将其转换为以点为中心的热图以进行最终预测。值得注意的是，所有原始MLLM参数都被冻结，确保其预训练能力的完整保留。实验表明，我们的设计在多个数据集上一致地提高了部件级定位精度，并且可以无缝集成到任何开源MLLM中。
英文摘要：
Visual grounding aims to associate free-form textual queries with specific regions in an image. While recent Multimodal Large Language Models (MLLMs) have demonstrated promising capabilities in this domain, they primarily excel at object-level grounding and often struggle with part-level grounding-an essential requirement for fine-grained tasks such as robotic manipulation. In this work, we introduce a general approach that equips any open-source MLLMs with accurate 2D part-level point grounding, offering a more direct alternative to conventional grounding representations. Our method leverages the attention mechanisms inherently present in MLLMs. By synthesizing text-conditioned, grounding-aware queries within intermediate layers via the proposed Q-Synth Module, we capture target-relevant attention patterns and refine them with a lightweight Attention-to-Point Decoder, which converts these patterns into a point-centric heatmap for final prediction. Notably, all original MLLM parameters are frozen, ensuring full preservation of their pre-trained capabilities. Experiments show that our design consistently improves part-level grounding accuracy across datasets and can be seamlessly integrated into any open-source MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
