CD-LoRA：面向多任务微调的一致性驱动低秩适配
CD-LoRA: Consistency-Driven Low-Rank Adaptation for Multi-Task Fine-Tuning
浏览论文内容
中文总结 AI 辅助
该研究针对多任务LoRA方法存在的训练-推理不一致问题，提出无路由的CD-LoRA，通过一致性驱动对齐机制提升多任务微调的稳定性与性能，优于现有多适配器基线。
中文摘要 AI 辅助
多任务学习（MTL）是将大语言模型（LLM）适配到不同领域的关键，但主流基于LoRA的方法依赖复杂的路由机制来划分任务特定知识。本研究通过二阶泰勒分析揭示了路由方差引发的不稳定性，指出这类基于路由的设计易出现训练-推理不一致问题，即分布偏移下的随机路由决策会损害推理稳定性。为此，我们提出一致性驱动低秩适配（CD-LoRA），完全移除路由模块，采用一致性驱动对齐机制在共享低秩空间中强制各任务表示一致，在无显式划分开销的情况下生成鲁棒的任务无关特征。大量实验表明，CD-LoRA在性能上始终优于最先进的多适配器基线，为多任务参数高效微调（PEFT）提供了更简单、无路由且更稳定的解决方案，代码可通过此匿名链接获取。
英文摘要
While Multi-Task Learning (MTL) is essential for adapting Large Language Models (LLMs) to diverse domains, prevailing LoRA-based methods rely on complex routing mechanisms that partition task-specific knowledge. In this work, we reveal that such routing-based designs are prone to a training-inference discrepancy, where stochastic routing decisions under distribution shifts compromise inference stability. Driven by a second-order Taylor analysis that exposes the instability induced by routing variance, we challenge the training-inference discrepancy and propose Consistency-Driven Low-Rank Adaptation (CD-LoRA). By eliminating routers entirely, CD-LoRA employs a consistency-driven alignment mechanism to enforce representation congruence across tasks in a shared low-rank space. This paradigm fosters robust, task-agnostic features without explicit partitioning overhead. Extensive experiments show that CD-LoRA consistently outperforms state-of-the-art multi-adapter baselines, offering a simpler, router-free, and more stable solution for multi-task PEFT. The code is available at the anonymous link https://github.com/zhaqian21/CD-LoRA.
发表机构
- School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）
- International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
机构由 AI 辅助整理，请以论文原文为准。
