MoDA: 面向指令型多模态大语言模型的细粒度视觉定位的调制适配器
MoDA: Modulation Adapter for Fine-Grained Visual Grounding in Instructional MLLMs
AI总结：
提出MoDA调制适配器，通过指令引导的通道级乘法调制增强细粒度视觉定位，在12个基准上对三种MLLM架构取得一致提升，计算开销极小。
摘要：
多模态大语言模型（MLLMs）通过将预训练的视觉编码器与大语言模型（LLMs）集成，在指令跟随任务中取得了显著成功。然而，现有方法由于视觉补丁表示中的语义纠缠，常常难以实现细粒度的视觉定位，其中单个补丁混合了多个不同的视觉元素，使得模型难以聚焦于指令相关的细节。为了应对这一挑战，我们提出了MoDA（调制适配器），一种轻量级模块，通过指令引导的通道级调制增强视觉定位。与Q-Former等执行加性特征选择的令牌级方法不同，MoDA通过对已对齐特征进行乘法调制在通道级操作，从而实现对每个指令相关嵌入维度的细粒度控制。遵循标准的LLaVA训练协议，MoDA在语言指令与预对齐的视觉特征之间应用交叉注意力，生成动态调制掩码，无需架构修改或额外监督。我们在涵盖视觉问答、视觉中心推理和幻觉检测的12个基准上评估了MoDA，包括最近的2024年基准（MMVP、CV-Bench、MMStar、RealWorldQA），并在三种不同的MLLM架构上进行了测试：LLaVA-1.5、LLaVA-MoRE（2025）和Qwen3-VL（2025）。MoDA在所有三个系列中均取得了一致的提升，在LLaVA-1.5系列的MMVP上提升了+12.0个百分点，在LLaVA-MoRE系列的ScienceQA上提升了+4.8个百分点，在Qwen3-VL上ScienceQA提升了+4.9、RealWorldQA提升了+4.1、GQA提升了+3.8，证实了这些增益在CLIP编码器之外具有泛化性，且计算开销极小（<1% FLOPs）。代码可在https://github.com/waybarrios/MoDA获取。
英文摘要：
Multimodal Large Language Models (MLLMs) have achieved remarkable success in instruction-following tasks by integrating pretrained visual encoders with large language models (LLMs). However, existing approaches often struggle with fine-grained visual grounding due to semantic entanglement in visual patch representations, where individual patches blend multiple distinct visual elements, making it difficult for models to focus on instruction-relevant details. To address this challenge, we propose MoDA (Modulation Adapter), a lightweight module that enhances visual grounding through instruction-guided channel-wise modulation. Unlike token-level methods such as Q-Former that perform additive feature selection, MoDA operates at the channel level through multiplicative modulation on already-aligned features, enabling fine-grained control over which embedding dimensions are relevant for each instruction. Following the standard LLaVA training protocol, MoDA applies cross-attention between language instructions and pre-aligned visual features, generating dynamic modulation masks without architectural modifications or additional supervision. We evaluate MoDA across 12 benchmarks spanning visual question answering, vision-centric reasoning, and hallucination detection, including recent 2024 benchmarks (MMVP, CV-Bench, MMStar, RealWorldQA), on three distinct MLLM architectures: LLaVA-1.5, LLaVA-MoRE (2025), and Qwen3-VL (2025). MoDA delivers consistent gains across all three families, with +12.0 points on MMVP for the LLaVA-1.5 family and +4.8 points on ScienceQA for the LLaVA-MoRE family, and +4.9 ScienceQA, +4.1 RealWorldQA, and +3.8 GQA on Qwen3-VL, confirming that the gains generalize beyond CLIP-based encoders with minimal overhead (<1% FLOPs). Code is available at https://github.com/waybarrios/MoDA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
