MoDE：利用二元专家混合实现高效的多任务参数高效微调
MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts
AI总结：
针对多任务 LoRA 混合中下投影矩阵冗余的问题，本文提出 MoDE，通过共享下投影、原子秩一适配器和任务路由器，在 SNI 700 多个任务上以不增参数实现优于现有 PEFT 方法的性能。
摘要：
诸如低秩适配（LoRA）之类的参数高效微调技术已经彻底改变了大型语言模型（LLM）针对多样化任务的适配方式。近期的研究探索了在多任务场景中使用 LoRA 模块的混合。然而，我们的分析揭示了这些架构中下投影矩阵存在冗余。这一观察促使我们提出了“二元专家混合”（Mixture of Dyadic Experts，MoDE）方法，该方法引入了一种用于高效多任务适配的新颖设计。具体而言，该方法通过在各任务之间共享下投影矩阵，并采用原子秩一适配器，同时结合能够实现更复杂任务级专门化的路由器来完成。我们的设计允许更细粒度的混合，从而增强模型联合处理多个任务的能力。我们在包含 700 多个多样化任务的 Supernatural Instructions（SNI）基准上评估了 MoDE，并证明其在不引入额外参数的情况下，性能优于最先进的多任务参数高效微调（PEFT）方法。我们的发现有助于更深入地理解多任务 LLM 适配中的参数效率，并为部署高性能、轻量级模型提供了实用的解决方案。
英文摘要：
Parameter-efficient fine-tuning techniques like Low-Rank Adaptation (LoRA) have revolutionized the adaptation of large language models (LLMs) to diverse tasks. Recent efforts have explored mixtures of LoRA modules for multi-task settings. However, our analysis reveals redundancy in the down-projection matrices of these architectures. This observation motivates our proposed method, Mixture of Dyadic Experts (MoDE), which introduces a novel design for efficient multi-task adaptation. This is done by sharing the down-projection matrix across tasks and employing atomic rank-one adapters, coupled with routers that allow more sophisticated task-level specialization. Our design allows for more fine-grained mixing, thereby increasing the model's ability to jointly handle multiple tasks. We evaluate MoDE on the Supernatural Instructions (SNI) benchmark consisting of a diverse set of 700+ tasks and demonstrate that it outperforms state-of-the-art multi-task parameter-efficient fine-tuning (PEFT) methods, without introducing additional parameters. Our findings contribute to a deeper understanding of parameter efficiency in multi-task LLM adaptation and provide a practical solution for deploying high-performing, lightweight models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
