InstMoE：具有专用专家的自适应多模态路由
InstMoE: Adaptive Multimodal Routing with Specialized Experts
另 1 家 · 查看机构详情
- Guangdong University of Technology（广东工业大学）
- University of Copenhagen（哥本哈根大学）
- Pengcheng Laboratory（鹏城实验室）
- Tongji University（同济大学）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
InstMoE提出自适应专家路由框架，通过动态选择单模态和跨模态专家并引入对比语义对齐，在多模态情感分析基准上以更少参数达到最先进性能。
中文摘要 AI 辅助
多模态输入本质上是异质的，不仅在不同模态之间存在差异，而且在有效预测所需的信息路径上也存在差异。为了解决这一局限性，我们提出了InstMoE，一种用于多模态学习的自适应专家路由框架。InstMoE将每个输入动态路由到专门的单模态和跨模态专家，使模型能够根据输入的特征调整其信息路径。然而，当模态特定的变化掩盖了任务相关的语义时，路由可能会被误导。这种不相关的变化可能扭曲路由决策，导致输入被分配给不合适的专家。因此，我们引入了对比语义对齐模块，该模块鼓励语义相似的输入共享任务相关的表示，同时抑制不相关的模态特定变化。在多模态情感分析基准上的实验表明，InstMoE在CMU-MOSEI和CH-SIMS v2上实现了最先进的性能，同时使用的参数远少于竞争性基线。进一步的分析表明，不同的输入表现出不同的专家偏好，证明InstMoE超越了固定融合，走向自适应多模态计算。
英文摘要
Multimodal inputs are inherently heterogeneous, not only across modalities but also in the information pathways required for effective prediction. To address this limitation, we propose InstMoE, an adaptive expert routing framework for multimodal learning. InstMoE dynamically routes each input to specialized unimodal and cross-modal experts, allowing the model to adapt its information pathways to the characteristics of the input. However, routing can be misled when modality-specific variations obscure task-relevant semantics. Such irrelevant variations may distort routing decisions, causing inputs to be assigned to inappropriate experts. We therefore introduce a Contrastive Semantic Alignment module, which encourages semantically similar inputs to share task-relevant representations while suppressing irrelevant modality-specific variations. Experiments on multimodal sentiment analysis benchmarks demonstrate that InstMoE achieves state-of-the-art performance on CMU-MOSEI and CH-SIMS v2 while using substantially fewer parameters than competitive baselines. Further analysis shows that different inputs exhibit distinct expert preferences, demonstrating that InstMoE moves beyond fixed fusion toward adaptive multimodal computation.
