Mixpert：利用高效的混合视觉专家机制缓解多模态学习冲突
Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts
AI总结：
提出 Mixpert 混合视觉专家架构，通过动态路由机制为输入图像分配最合适的视觉专家，有效缓解了多模态大语言模型中单一视觉编码器的多任务领域冲突，并在极少额外计算成本下实现显著性能提升。
摘要：
多模态大语言模型（MLLMs）需要对复杂的图像信息进行细致的解读，通常利用视觉编码器来感知各种视觉场景。然而，仅依靠单一视觉编码器来处理多样的任务领域十分困难，且不可避免地会导致冲突。近期的研究通过直接整合多个特定领域的视觉编码器来增强数据感知能力，但这种结构增加了复杂性，并限制了联合优化的潜力。在本文中，我们提出了 Mixpert，这是一种高效的混合视觉专家架构，它继承了单一视觉编码器的联合学习优势，同时被重构为多专家范式，以便在不同视觉任务中进行特定任务的微调。此外，我们设计了一种动态路由机制，将输入图像分配给最合适的视觉专家。Mixpert 以极少的额外计算成本，有效缓解了单一视觉编码器在多任务学习中遇到的领域冲突，使其比使用多个编码器更高效。此外，Mixpert 可以无缝集成到任何 MLLM 中，实验结果表明其在各种任务上均取得了显著的性能提升。
英文摘要：
Multimodal large language models (MLLMs) require a nuanced interpretation of complex image information, typically leveraging a vision encoder to perceive various visual scenarios. However, relying solely on a single vision encoder to handle diverse task domains proves difficult and inevitably leads to conflicts. Recent work enhances data perception by directly integrating multiple domain-specific vision encoders, yet this structure adds complexity and limits the potential for joint optimization. In this paper, we introduce Mixpert, an efficient mixture-of-vision-experts architecture that inherits the joint learning advantages from a single vision encoder while being restructured into a multi-expert paradigm for task-specific fine-tuning across different visual tasks. Additionally, we design a dynamic routing mechanism that allocates input images to the most suitable visual expert. Mixpert effectively alleviates domain conflicts encountered by a single vision encoder in multi-task learning with minimal additional computational cost, making it more efficient than multiple encoders. Furthermore, Mixpert integrates seamlessly into any MLLM, with experimental results demonstrating substantial performance gains across various tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
