AsyMoE：利用模态不对称性提升大视觉-语言模型专家专业化
AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
AI总结：
AsyMoE通过三个专门专家组解决视觉-语言模态不对称问题，提升大模型专家专业化性能。
摘要：
大型视觉-语言模型（LVLMs）通过扩展的架构和广泛的训练在多模态任务中表现出色。然而，现有的专家混合（MoE）方法由于视觉和语言处理之间的不对称性而面临挑战。视觉信息是空间完整的，而语言需要保持序列上下文。因此，MoE模型在平衡模态特定特征和跨模态交互方面存在困难。通过系统分析，我们发现语言专家在更深的层中逐渐失去上下文基础，并更多依赖参数知识而不是利用提供的视觉和语言信息。为了解决这个问题，我们提出了AsyMoE，一种新的架构，通过三个专门的专家组来建模这种不对称性。我们设计了模态特定的专家用于模态特定处理，双曲的跨模态专家用于分层的跨模态交互，以及证据优先的语言专家以抑制参数偏差并保持上下文基础。广泛的实验表明，AsyMoE在vanilla MoE和模态特定MoE上分别实现了26.58%和15.45%的准确率提升，同时激活的参数比密集模型少25.45%。
英文摘要：
Large Vision-Language Models (LVLMs) have demonstrated impressive performance on multimodal tasks through scaled architectures and extensive training. However, existing Mixture of Experts (MoE) approaches face challenges due to the asymmetry between visual and linguistic processing. Visual information is spatially complete, while language requires maintaining sequential context. As a result, MoE models struggle to balance modality-specific features and cross-modal interactions. Through systematic analysis, we observe that language experts in deeper layers progressively lose contextual grounding and rely more on parametric knowledge rather than utilizing the provided visual and linguistic information. To address this, we propose AsyMoE, a novel architecture that models this asymmetry using three specialized expert groups. We design intra-modality experts for modality-specific processing, hyperbolic inter-modality experts for hierarchical cross-modal interactions, and evidence-priority language experts to suppress parametric biases and maintain contextual grounding. Extensive experiments demonstrate that AsyMoE achieves 26.58% and 15.45% accuracy improvements over vanilla MoE and modality-specific MoE respectively, with 25.45% fewer activated parameters than dense models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
