MACS: 模态感知容量缩放用于高效多模态MoE推理
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
AI总结：
针对多模态MoE大模型在专家并行推理中因信息异质性和模态动态性导致的效率瓶颈，提出无需训练的MACS框架，通过熵加权负载和动态模态自适应容量机制优化资源分配，显著提升多模态基准性能。
摘要：
混合专家多模态大语言模型（MoE MLLMs）在专家并行（EP）推理过程中因落后者效应而遭受显著的效率瓶颈。在多模态背景下，这一问题更加严重，因为现有的基于token计数的负载均衡方法无法解决两个独特挑战：（1）信息异质性，其中大量冗余的视觉token与语义关键的token被同等对待；（2）模态动态性，不同任务中视觉与文本比例的变化导致资源错配。为应对这些挑战，我们提出MACS（模态感知容量缩放），一种无需训练的推理框架。具体而言，MACS引入熵加权负载机制来量化视觉token的语义价值，解决信息异质性。此外，动态模态自适应容量机制根据输入的实时模态组成分配专家资源。大量实验表明，MACS在各种多模态基准上显著优于现有方法，为MoE MLLMs在EP推理中的高效部署提供了新颖且稳健的解决方案。
英文摘要：
Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect. This issue is worsened in the multimodal context, as existing token-count-based load balancing methods fail to address two unique challenges: (1) Information Heterogeneity, where numerous redundant visual tokens are treated equally to semantically critical ones, and (2) Modality Dynamics, where varying visual to text ratios across tasks lead to resource misallocation. To address these challenges, we propose MACS (Modality-Aware Capacity Scaling), a training-free inference framework. Specifically, MACS introduces an Entropy-Weighted Load mechanism to quantify the semantic value of visual tokens, addressing information heterogeneity. Additionally, the Dynamic Modality-Adaptive Capacity mechanism allocates expert resources based on the real-time modal composition of the input. Extensive experiments demonstrate that MACS significantly outperforms existing methods on various multimodal benchmarks, providing a novel and robust solution for the efficient deployment of MoE MLLMs in EP inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
