arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2508.08066cs.CVcs.AIcs.CLcs.LG

ExpVG: 探索多模态大语言模型中视觉定位的设计空间

ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model

University of Illinois Chicago(伊利诺伊大学香槟分校);Sony AI(索尼人工智能)

Weitai Kang, Weiming Zhuang, Zhizhong Li, Yan Yan, Lingjuan Lyu

AI总结:

针对多模态大语言模型(MLLMs)视觉定位(VG)任务中设计选择缺乏系统验证的问题,本文以LLaVA-1.5为基础,全面研究影响VG性能的设计选择,优化模型并在RefCOCO系列数据集上取得显著提升。

摘要:

多模态大语言模型(MLLMs)的细粒度多模态能力已成为关键研究方向,尤其针对视觉定位(VG)问题。尽管现有方法性能优异,但在为VG微调MLLMs时,常采用不同设计选择,缺乏系统验证支持。为填补这一空白,本文全面研究影响MLLMs视觉定位性能的各类设计选择。我们使用被广泛用于先前实证研究的LLaVA-1.5进行分析;虽存在更新模型,但遵循该惯例以确保研究结果广泛适用且可扩展至其他架构。研究涵盖两大关键方面:(1)探索MLLMs中不同视觉定位范式,识别最有效设计并提供见解;(2)对定位数据设计进行消融研究,优化MLLMs针对VG任务的微调。最终,研究成果助力构建更强的VG专用MLLM,在RefCOCO/+/g数据集上较LLaVA-1.5分别提升5.6%、6.9%和7.0%。

英文摘要:

Fine-grained multimodal capability in Multimodal Large Language Models (MLLMs) has emerged as a critical research direction, particularly for tackling the visual grounding (VG) problem. Despite the strong performance achieved by existing approaches, they often employ disparate design choices when fine-tuning MLLMs for VG, lacking systematic verification to support these designs. To bridge this gap, this paper presents a comprehensive study of various design choices that impact the VG performance of MLLMs. We conduct our analysis using LLaVA-1.5, which has been widely adopted in prior empirical studies of MLLMs. While more recent models exist, we follow this convention to ensure our findings remain broadly applicable and extendable to other architectures. We cover two key aspects: (1) exploring different visual grounding paradigms in MLLMs, identifying the most effective design, and providing our insights; and (2) conducting ablation studies on the design of grounding data to optimize MLLMs' fine-tuning for the VG task. Finally, our findings contribute to a stronger MLLM for VG, achieving improvements of +5.6% / +6.9% / +7.0% on RefCOCO/+/g over the LLaVA-1.5.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑