通过语言引导的语义线索实现对遮挡和小物体的鲁棒接地
Robust Grounding with MLLMs Against Occlusion and Small Objects via Language-Guided Semantic Cues
AI总结：
本文提出通过语言引导的语义线索提升MLLM在拥挤场景中的接地能力，通过语义线索提取器和文本嵌入引导优化对象语义，实验表明能有效提高接地精度。
摘要：
尽管多模态大语言模型（MLLMs）在一般场景中增强了接地能力，但其在拥挤场景中的鲁棒性仍待探索。拥挤场景存在视觉挑战（即遮挡和小物体），这些挑战会损害对象语义并降低接地性能。相比之下，语言表达不受此类降级影响，能保持对象语义。基于这些观察，我们提出了一种新的方法，通过利用语言引导的语义线索（LGSCs）来克服这些限制。具体而言，我们的方法引入了一个语义线索提取器（SCE），从MLLM的视觉管道中推导出对象的语义线索。然后，通过相应的文本嵌入引导这些线索，生成作为语言语义先验的LGSCs。随后，它们被重新整合到原始视觉管道中以细化对象语义。广泛的实验和分析表明，将LGSCs整合到MLLM中能有效提高拥挤场景中的接地准确性。
英文摘要：
While Multimodal Large Language Models (MLLMs) have enhanced grounding capabilities in general scenes, their robustness in crowded scenes remains underexplored. Crowded scenes entail visual challenges (i.e., occlusion and small objects), which impair object semantics and degrade grounding performance. In contrast, language expressions are immune to such degradation and preserve object semantics. In light of these observations, we propose a novel method that overcomes such constraints by leveraging Language-Guided Semantic Cues (LGSCs). Specifically, our approach introduces a Semantic Cue Extractor (SCE) to derive semantic cues of objects from the visual pipeline of an MLLM. We then guide these cues using corresponding text embeddings to produce LGSCs as linguistic semantic priors. Subsequently, they are reintegrated into the original visual pipeline to refine object semantics. Extensive experiments and analyses demonstrate that incorporating LGSCs into an MLLM effectively improves grounding accuracy in crowded scenes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
