R2-T2：面向多模态混合专家模型的测试时重路由
R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
AI总结：
针对大型多模态模型中端到端路由器无法为测试样本生成最优权重的问题，提出测试时重路由（R2-T2）方法，通过局部优化路由权重向量，在无需训练基础模型参数的情况下显著提升模型在多任务基准上的性能。
摘要：
在大型多模态模型（LMMs）中，非语言模态（如视觉表示）的感知能力通常无法与大型语言模型（LLMs）强大的推理能力相媲美，这阻碍了 LMMs 在具有挑战性的下游任务上的表现。最近，通过用混合专家模型（MoE）替换视觉编码器，这一问题得到了缓解，因为 MoE 能够提供多样化下游任务所需的丰富、多粒度且多样化的表示。多模态 MoE 的性能在很大程度上取决于其路由器，该路由器为每个输入重新加权并混合不同专家的表示。然而，我们发现端到端训练的路由器并不总是为每个测试样本产生最优的路由权重。为了弥合这一差距，我们提出了一种新颖且高效的方法“测试时重路由（R2-T2）”，该方法在测试时通过将路由权重向量向测试样本邻域内正确预测样本的权重向量移动，来局部优化该向量。我们提出了三种具有不同优化目标和邻域搜索空间的 R2-T2 策略。R2-T2 在无需训练任何基础模型参数的情况下，持续且大幅地提升了最先进 LMMs 在多样化任务挑战性基准上的性能。
英文摘要：
In large multimodal models (LMMs), the perception of non-language modalities (e.g., visual representations) is usually not on par with the large language models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on challenging downstream tasks. This weakness has been recently mitigated by replacing the vision encoder with a mixture-of-experts (MoE), which provides rich, multi-granularity, and diverse representations required by diverse downstream tasks. The performance of multimodal MoE largely depends on its router, which reweights and mixes the representations of different experts for each input. However, we find that the end-to-end trained router does not always produce the optimal routing weights for every test sample. To bridge the gap, we propose a novel and efficient method "Re-Routing in Test-Time (R2-T2)" that locally optimizes the vector of routing weights in test-time by moving it toward those vectors of the correctly predicted samples in a neighborhood of the test sample. We propose three R2-T2 strategies with different optimization objectives and neighbor-search spaces. R2-T2 consistently and greatly improves state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks, without training any base-model parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
