迈向LLM的下一个前沿：在私有数据上训练：一个跨域基准用于联邦微调
Towards the Next Frontier of LLMs, Training on Private Data: A Cross-Domain Benchmark for Federated Fine-Tuning
AI总结：
本文提出通过联邦学习平台在跨域基准中评估联邦微调方法，展示在隐私数据上训练LLM的可行性，对比三种参数高效微调策略，证明联邦微调在非独立同分布数据下的有效性。
摘要：
近年来，大语言模型（LLMs）的成功主要归功于庞大的公开数据集。然而，LLM的发展下一步将超越公开数据。世界最宝贵的信息大多是私有的，尤其是在医疗和金融等受严格监管的领域，其中的数据包括患者病史或客户通信。解锁这些数据可能代表重大进展，使LLM具备更深入的领域专业知识和更强的实际应用价值。然而，这些数据无法共享，因为它们分布在机构中，并受到隐私、法规和组织障碍的限制。此外，机构数据通常是非独立同分布（non-IID），在不同地点存在人口特征、数据模态、文档模式和任务特定标签分布的差异。本文展示了一种实用方法，通过跨数据孤岛的联邦合作来解锁私有和分布的机构数据，用于LLM的适应。基于Sherpa.ai联邦学习平台，我们的框架使节点能够联合微调共享的LLM而无需交换私有数据。我们通过医疗和金融的跨域基准评估了这种方法，使用四个封闭式问答和分类数据集：MedQA、MedMCQA、FPB和FiQA-SA。我们比较了三种参数高效微调（PEFT）策略——LoRA、QLoRA和IA3——在非IID设置下，反映机构数据异质性的预训练背骨。我们的结果表明，联邦微调在非IID数据下表现接近集中训练，并优于孤立单机构学习。从绿色AI的角度来看，QLoRA和IA3在有限的准确率下降下提高了效率，支持联邦PEFT作为在数据无法共享时适应LLM的可行方法。
英文摘要：
The recent success of large language models (LLMs) has been largely driven by vast public datasets. However, the next frontier for LLM development lies beyond public data. Much of the world's most valuable information is private, especially in highly regulated sectors such as healthcare and finance, where data include patient histories or customer communications. Unlocking this data could represent a major leap forward, enabling LLMs with deeper domain expertise and stronger real-world utility. Yet, these data cannot be shared because they are distributed across institutions and constrained by privacy, regulatory, and organizational barriers. Moreover, institutional datasets are typically non-independent and identically distributed (non-IID), differing across sites in population characteristics, data modalities, documentation patterns, and task-specific label distributions. In this paper, we demonstrate a practical approach to unlocking private and distributed institutional data for LLM adaptation through federated collaboration across data silos. Built on the Sherpa.ai Federated Learning platform, our framework enables nodes to jointly fine-tune a shared LLM without exchanging private data. We evaluate this approach through a cross-domain benchmark in healthcare and finance, using four closed-ended question answering and classification datasets: MedQA, MedMCQA, FPB, and FiQA-SA. We compare three parameter-efficient fine-tuning (PEFT) strategies-LoRA, QLoRA, and IA3-across pretrained backbones under non-IID settings reflecting institutional data heterogeneity. Our results show that federated fine-tuning performs close to centralized training and outperforms isolated single-institution learning. From a Green AI perspective, QLoRA and IA3 improve efficiency with limited accuracy degradation, supporting federated PEFT as a viable approach for adapting LLMs where data cannot be shared.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
