关于令牌的困境：具有漂移感知令牌分配的动态MoE用于大视觉语言模型的持续学习
On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
AI总结：
本文提出LLaVA-DyMoE框架，通过动态MoE和漂移感知令牌分配缓解持续学习中的路由漂移问题，提升模型在持续学习中的性能。
摘要：
多模态持续指令微调旨在通过学习新数据持续增强大视觉语言模型（LVLMs），同时不遗忘之前获得的知识。混合专家（MoE）架构通过逐步添加新专家并扩展路由器，同时保持现有专家冻结，自然促进这一过程。然而，尽管专家隔离，基于MoE的持续学习者仍因路由漂移而遗忘：旧任务的令牌被错误地吸引到新添加的专家，从而降低先前任务的性能。我们分析了令牌层面的失败模式，并揭示了令牌的困境：新任务数据中的模糊和旧令牌提供最小的学习收益，但当路由到新专家时会引发遗忘，这归因于训练期间的模糊路由分配。受此启发，我们提出LLaVA-DyMoE，一种动态MoE框架，通过具有漂移感知的令牌分配逐步扩展MoE。我们通过路由分数分布表征令牌类型，并应用针对性的正则化。具体而言，令牌层面的分配指导将模糊和旧令牌引导远离新专家，以保持已建立的路由模式并减轻路由漂移，同时互补的路由分数正则化强制专家组分离并促进新专家的专业化。广泛的实验表明，我们的LLaVA-DyMoE有效缓解了由路由漂移引起的遗忘，相比基线模型，在平均最终准确率上提高了超过7%，在遗忘减少方面减少了12%。项目页面是https://zhaoc5.github.io/DyMoE。
英文摘要：
Multimodal Continual Instruction Tuning aims to continually enhance Large Vision Language Models (LVLMs) by learning from new data without forgetting previously acquired knowledge. Mixture of Experts (MoE) architectures naturally facilitate this by incrementally adding new experts and expanding routers while keeping the existing ones frozen. However, despite expert isolation, MoE-based continual learners still suffer from forgetting due to routing-drift: old-task tokens become mistakenly attracted to newly added experts, degrading performance on prior tasks. We analyze the failure mode at the token level and reveal the token's dilemma: ambiguous and old tokens in new-task data offer minimal learning benefit yet induce forgetting when routed to new experts, due to their ambiguous routing assignment during training. Motivated by this, we propose LLaVA-DyMoE, a dynamic MoE framework that incrementally expands the MoE with drift-aware token assignment. We characterize token types via their routing score distributions and apply targeted regularization. Specifically, a token-level assignment guidance steers ambiguous and old tokens away from new experts to preserve established routing patterns and alleviate routing-drift, while complementary routing score regularizations enforce expert-group separation and promote new-expert specialization. Extensive experiments demonstrate that our LLaVA-DyMoE effectively mitigates routing-drift-induced forgetting, achieving over a 7% gain in mean final accuracy and a 12% reduction in forgetting compared to baselines. The project page is https://zhaoc5.github.io/DyMoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
