arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.06954cs.LGcs.CLcs.DCcs.MA
收藏

OpenFedLLM:通过联邦学习在去中心化私有数据上训练大型语言模型

OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室)

Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, Siheng Chen

AI总结:

本文提出OpenFedLLM框架,通过联邦学习在去中心化私有数据上训练大型语言模型,涵盖联邦指令微调、价值对齐及7种FL算法,实验证明其优于本地训练且微调后的Llama2-7B在金融基准上超越GPT-4。

摘要:

在大量公开可用数据上训练的大型语言模型(LLMs)在各个领域展现了巨大的成功。虽然更多的数据有助于提升性能,但一个令人不安的现实是,高质量的公开数据将在几年内耗尽。在本文中,我们为当代LLMs提供了一个潜在的下一步:通过联邦学习(FL)在未被充分利用的分布式私有数据上进行协作且保护隐私的LLM训练,其中多个数据所有者在不传输原始数据的情况下协作训练一个共享模型。为了实现这一目标,我们构建了一个简洁、集成且便于研究的框架/代码库,命名为OpenFedLLM。它涵盖了用于增强指令遵循能力的联邦指令微调、用于与人类价值观对齐的联邦价值对齐,以及7种代表性的FL算法。此外,OpenFedLLM支持在多个领域进行训练,涵盖了8个训练数据集;并提供了全面的评估,涵盖了30多个评估指标。通过大量实验,我们观察到所有FL算法在LLM训练上均优于本地训练,展示了在各种设置下明显的性能提升。值得注意的是,在一个金融基准测试中,通过应用任何FL算法微调的Llama2-7B都能以显著优势超越GPT-4,而通过单独训练获得的模型则不能,这为客户端参与FL提供了强大的动力。代码可在https://github.com/rui-ye/OpenFedLLM获取。

英文摘要:

Trained on massive publicly available data, large language models (LLMs) have demonstrated tremendous success across various fields. While more data contributes to better performance, a disconcerting reality is that high-quality public data will be exhausted in a few years. In this paper, we offer a potential next step for contemporary LLMs: collaborative and privacy-preserving LLM training on the underutilized distributed private data via federated learning (FL), where multiple data owners collaboratively train a shared model without transmitting raw data. To achieve this, we build a concise, integrated, and research-friendly framework/codebase, named OpenFedLLM. It covers federated instruction tuning for enhancing instruction-following capability, federated value alignment for aligning with human values, and 7 representative FL algorithms. Besides, OpenFedLLM supports training on diverse domains, where we cover 8 training datasets; and provides comprehensive evaluations, where we cover 30+ evaluation metrics. Through extensive experiments, we observe that all FL algorithms outperform local training on training LLMs, demonstrating a clear performance improvement across a variety of settings. Notably, in a financial benchmark, Llama2-7B fine-tuned by applying any FL algorithm can outperform GPT-4 by a significant margin while the model obtained through individual training cannot, demonstrating strong motivation for clients to participate in FL. The code is available at https://github.com/rui-ye/OpenFedLLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑