超越编码器累加：衡量多编码器视觉语言模型中编码器的作用
Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
AI总结：
通过重新训练所有31个非空子集，提出容量-必要性分解和预投影器秩分析，揭示多编码器视觉语言模型中编码器角色并非简单累加，并给出最优配对原则。
摘要：
随着基础模型向融合更多异构视觉流扩展，理解不同编码器在联合训练下的交互成为原则性设计的前提。然而，大型视觉语言模型目前缺乏相应的工具，且参数高效的编码器配置在训练前难以识别。为了重新审视联合训练下的编码器角色，我们在16基准的Cambrian-1套件上，在统一流程下重新训练并评估了五个常见视觉编码器的所有31个非空子集（总计约2万GPU小时），并报告了三个发现。首先，从头重新训练每个子集揭示了与在固定检查点上掩码编码器所得不同的编码器排名，包括哪个编码器整体排名第一。其次，我们将每个编码器的贡献分解为两个维度：容量（编码器自身达到的分数）和必要性（从完整池中移除时的下降）。这两个维度不可互换。配对两个最高容量的编码器是次优的，而将一个高容量锚点与一个自适应补充配对则匹配完整的五编码器模型。在此配对之外添加更多编码器仅带来边际收益。第三，在固定参数数量下，每个编码器的预投影器有效秩解释了残差分数变化。最强的配对结合了一个秩在联合训练中存活的锚点和一个秩在联合训练下扩展的补充，这表明更高秩、更少坍缩的投影器输入对应着编码器-投影器接口处更有利的优化机制。总之，容量-必要性分解和预投影器秩分析，连同通过重新训练进行的全面评估，揭示了多编码器视觉语言模型设计中的方法论差距，并提供了弥补这一差距的具体原语。
英文摘要：
As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large vision-language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline (~20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder's contribution into two axes, Capacity, the score an encoder reaches on its own, and Necessity, the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank expands under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder-projector interface. Together, the Capacity-Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
