LoRA融合能否支持云边协作中的跨域任务?
Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?
AI总结:
针对云边协作中跨域问题解决的需求,提出剪枝-训练-恢复框架和冲突解决模块LoRA-CR,发现现有LoRA融合方法在跨域基准MMLU-CD上表现不佳,而LoRA-CR通过缓解参数冲突将性能提升高达3.8%。
摘要:
云托管的大型语言模型(LLM)通常依赖LoRA进行领域自适应,但领域数据分布在多个边缘设备上,且由于隐私约束无法上传。这引发了一个基本问题:如何将多个私有边缘的知识整合到云LLM中以解决跨域问题?一个自然的解决方案是本地训练LoRA适配器并在云端融合;然而,现有流水线依赖于不切实际的假设,即边缘设备可以托管云规模的LLM,并且主要在单域任务上进行评估。为了解决这些限制,我们提出了一个剪枝-训练-恢复框架,支持在剪枝模型上进行本地LoRA训练和隐私保护的云集成。我们进一步引入了MMLU-CD,一个跨域基准,将多个领域样本组合成单个实例,从而能够显式评估跨域问题解决能力。这使我们能够提出一个具体问题:现有的LoRA融合方法能否支持云边协作中的跨域任务?我们的实证答案是否定的。现有的LoRA融合方法在MMLU-CD上表现不佳,通常不如基础LLM,揭示了它们无法支持跨域问题解决。我们将这一失败归因于LoRA适配器之间的参数冲突,并提出了一个简单的冲突解决模块LoRA-CR,它缓解了冲突更新,并将LoRA融合性能提升了高达3.8%。这些结果将冲突缓解确定为云边LoRA融合中一个关键但很大程度上被忽视的因素,值得在未来的研究中进一步探讨。
英文摘要:
Cloud-hosted large language models (LLMs) commonly rely on LoRA for domain adaptation, yet domain data are distributed across multiple edge devices and cannot be uploaded due to privacy constraints. This raises a fundamental question: how can knowledge from multiple private edges be integrated into a cloud LLM for cross-domain problem solving? A natural solution is to train LoRA adapters locally and fuse them in the cloud; however, existing pipelines rely on unrealistic assumptions that edge devices can host cloud-scale LLMs and are evaluated mainly on single-domain tasks. To address these limitations, we propose a prune-train-recover framework that enables local LoRA training on pruned models and privacy-preserving cloud integration. We further introduce MMLU-CD, a cross-domain benchmark that composes multiple domain samples into a single instance, enabling explicit evaluation of cross-domain problem solving. This allows us to ask a concrete question: Can existing LoRA fusion methods support cross-domain tasks in cloud-edge collaboration? Our empirical answer is negative. Existing LoRA fusion methods perform poorly on MMLU-CD, often underperforming the base LLM, revealing their inability to support cross-domain problem solving. We attribute this failure to parameter conflicts among LoRA adapters and propose a simple conflict-resolution module, LoRA-CR, which mitigates conflicting updates and improves LoRA fusion performance by up to 3.8%. These results identify conflict mitigation as a critical yet largely overlooked factor in cloud-edge LoRA fusion, warranting further investigation in future research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。