MoSLD：一种用于多任务学习的参数极其高效的共享低秩适配混合模型
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
AI总结：
针对LoRA多任务学习性能不足、MoE架构存在数据干扰、知识遗忘及参数量大的问题，提出带dropout策略的共享LoRA混合模型MoSLD，通过共享上投影矩阵学通用知识、下投影矩阵学任务特征，兼具优异的单多任务性能与强域外泛化能力。
摘要：
近年来，LoRA（低秩适配）已成为微调大型预训练模型的关键技术，但其在多任务学习场景中的性能往往不尽如人意。相比之下，MoE（混合专家）架构为这一问题提供了天然的解决方案，但它也带来了多领域数据相互干扰、不同任务知识遗忘等挑战。此外，MoE会大幅增加参数量，带来计算成本方面的难题。因此，本文提出了MoSLD，一种带有dropout策略的共享LoRA混合模型。MoSLD通过在不同专家之间共享LoRA的上投影矩阵来应对这些挑战，促使模型学习跨任务的通用知识，同时仍允许下投影矩阵聚焦于每个任务的独特特征。dropout的应用缓解了参数矩阵更新不平衡的问题，也减轻了LoRA中的参数过拟合现象。大量实验表明，该模型在单任务和多任务场景下均表现出优异性能，具备强大的域外泛化能力。
英文摘要：
Recently, LoRA has emerged as a crucial technique for fine-tuning large pre-trained models, yet its performance in multi-task learning scenarios often falls short. In contrast, the MoE architecture presents a natural solution to this issue. However, it introduces challenges such as mutual interference of data across multiple domains and knowledge forgetting of various tasks. Additionally, MoE significantly increases the number of parameters, posing a computational cost challenge. Therefore, in this paper, we propose MoSLD, a mixture-of-shared-LoRAs model with a dropout strategy. MoSLD addresses these challenges by sharing the upper projection matrix in LoRA among different experts, encouraging the model to learn general knowledge across tasks, while still allowing the lower projection matrix to focus on the unique features of each task. The application of dropout alleviates the imbalanced update of parameter matrix and mitigates parameter overfitting in LoRA. Extensive experiments demonstrate that our model exhibits excellent performance in both single-task and multi-task scenarios, with robust out-of-domain generalization capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
