DeLo：双分解低秩专家协作用于持续缺失模态学习
DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning
AI总结：
DeLo通过双分解低秩专家架构解决持续缺失模态学习中的模态干扰问题，结合跨模态引导路由和任务键记忆实现高效推理。
摘要：
适应大规模多模态模型（LMMs）到现实世界场景面临双重挑战：在处理频繁模态不完整性的过程中学习序列数据流，这被称为持续缺失模态学习（CMML）。然而，现有CMML工作主要依赖于提示调优，这种技术由于其可学习提示在共享嵌入空间中的跨任务干扰而难以处理此任务。简单应用低秩适应（LoRA）与模态共享模块也会因竞争梯度产生模态干扰。为此，我们提出了DeLo，首个利用新型双分解低秩专家架构进行CMML的框架。具体而言，该架构通过分解的LoRA专家解决模态干扰，动态组合LoRA更新矩阵与从解耦的模态特定因子池中提取的秩一因子。嵌入在结构上防止灾难性遗忘的任务划分框架中，该专家系统由两个关键机制支持：跨模态引导路由策略以处理不完整数据，以及任务键记忆以实现高效、任务无关的推理。在已建立的CMML基准上的大量实验表明，我们的方法显著优于最先进的方法。这突显了在现实多模态挑战中，系统化、架构意识的LoRA设计的价值。
英文摘要：
Adapting Large Multimodal Models (LMMs) to real-world scenarios poses the dual challenges of learning from sequential data streams while handling frequent modality incompleteness, a task known as Continual Missing Modality Learning (CMML). However, existing works on CMML have predominantly relied on prompt tuning, a technique that struggles with this task due to cross-task interference between its learnable prompts in their shared embedding space. A naive application of Low-Rank Adaptation (LoRA) with modality-shared module will also suffer modality interference from competing gradients. To this end, we propose DeLo, the first framework to leverage a novel dual-decomposed low-rank expert architecture for CMML. Specifically, this architecture resolves modality interference through decomposed LoRA expert, dynamically composing LoRA update matrix with rank-one factors from disentangled modality-specific factor pools. Embedded within a task-partitioned framework that structurally prevents catastrophic forgetting, this expert system is supported by two key mechanisms: a Cross-Modal Guided Routing strategy to handle incomplete data and a Task-Key Memory for efficient, task-agnostic inference. Extensive experiments on established CMML benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches. This highlights the value of a principled, architecturally-aware LoRA design for real-world multimodal challenges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
