探究多模态大语言模型中多个视觉编码器的冗余性
Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
AI总结：
本文研究了多模态大语言模型中多个视觉编码器的冗余性，通过分析发现编码器的专业化、冗余性和有害性，并提出了量化冗余的指标，为更高效的多模态架构设计提供依据。
摘要：
近期的多模态大语言模型（MLLMs）越来越多地整合多个视觉编码器以提高在各种基准测试中的性能，假设不同的预训练目标会产生互补的视觉信号。然而，我们显示这一假设在实践中往往不成立。通过在代表性多编码器MLLMs中系统地遮蔽编码器，我们发现当选择的编码器被遮蔽时，性能通常会优雅地下降，有时甚至会提高，揭示了普遍存在的编码器冗余。为了量化这种影响，我们引入了两个原理性指标：条件利用率率（CUR），它衡量编码器在存在其他编码器时的边际贡献，以及信息间隙（IG），它捕捉模型中编码器效用的异质性。使用这些工具，我们观察到：（i）在OCR和图表等任务上存在强烈的专业化，其中单个编码器可以主导，其CUR大于90%；（ii）在通用VQA和基于知识的任务上存在高冗余，其中编码器基本上可以互换；（iii）存在有害的编码器，其CUR为负数。值得注意的是，遮蔽特定编码器可以在特定任务类别上获得高达16%的更高准确性，比完整模型高出3.6%的整体性能提升。此外，单编码器和双编码器变体在大多数非OCR任务上恢复了超过90%的基线，使用显著更低的训练资源和推理延迟。我们的分析挑战了多编码器更好这一启发式方法，并为开发更高效和有效的多模态架构提供了可行的诊断方法。
英文摘要：
Recent multimodal large language models (MLLMs) increasingly integrate multiple vision encoders to improve performance on various benchmarks, assuming that diverse pretraining objectives yield complementary visual signals. However, we show this assumption often fails in practice. Through systematic encoder masking across representative multi encoder MLLMs, we find that performance typically degrades gracefully, and sometimes even improves, when selected encoders are masked, revealing pervasive encoder redundancy. To quantify this effect, we introduce two principled metrics: the Conditional Utilization Rate (CUR), which measures an encoder s marginal contribution in the presence of others, and the Information Gap (IG), which captures heterogeneity in encoder utility within a model. Using these tools, we observe: (i) strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater than 90 percent, (ii) high redundancy on general VQA and knowledge based tasks, where encoders are largely interchangeable, (iii) instances of detrimental encoders with negative CUR. Notably, masking specific encoders can yield up to 16 percent higher accuracy on a specific task category and 3.6 percent overall performance boost compared to the full model.Furthermore, single and dual encoder variants recover over 90 percent of baseline on most non OCR tasks with substantially lower training resources and inference latency. Our analysis challenges the more encoders are better heuristic in MLLMs and provides actionable diagnostics for developing more efficient and effective multimodal architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
