arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.04845cs.CLcs.AIcs.DCcs.LGcs.MA
收藏

FedLLM-Bench:面向大语言模型联邦学习的现实基准

FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models

Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Shanghai AI Laboratory(上海人工智能实验室)

Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, Siheng Chen

AI总结:

本文提出FedLLM-Bench,一个包含8种训练方法、4个数据集和6项评估指标的大语言模型联邦学习现实基准,覆盖指令微调与偏好对齐,客户端规模达38至747,旨在提供实用测试平台并促进公平比较。

摘要:

联邦学习已使多方能够在不直接共享数据的情况下协作训练大语言模型(FedLLM)。遵循这一训练范式,社区在框架、性能和隐私等多个方面投入了大量努力。然而,一个令人不快的事实是,目前尚不存在面向FedLLM的现实数据集和基准,此前的工作均依赖人工构造的数据集,未能捕捉真实世界场景中的特性。针对这一问题,我们提出了FedLLM-Bench,其中包含8种训练方法、4个训练数据集和6项评估指标,旨在为FedLLM社区提供一个全面的测试平台。FedLLM-Bench涵盖三个用于联邦指令微调的数据集(例如用户标注的多语言数据集)和一个用于联邦偏好对齐的数据集(例如用户标注的偏好数据集),其客户端数量规模从38到747不等。我们的数据集融合了多种具有代表性的多样性:语言、质量、数量、指令、长度、嵌入和偏好,从而捕捉真实世界场景中的特性。基于FedLLM-Bench,我们在所有数据集上进行了实验,以对现有联邦学习方法进行基准测试,并提供实证见解(例如多语言协作)。我们相信,我们的FedLLM-Bench能够通过减少所需工作、提供实用测试平台和促进公平比较,使FedLLM社区受益。代码和数据集可在https://github.com/rui-ye/FedLLM-Bench获取。

英文摘要:

Federated learning has enabled multiple parties to collaboratively train large language models without directly sharing their data (FedLLM). Following this training paradigm, the community has put massive efforts from diverse aspects including framework, performance, and privacy. However, an unpleasant fact is that there are currently no realistic datasets and benchmarks for FedLLM and previous works all rely on artificially constructed datasets, failing to capture properties in real-world scenarios. Addressing this, we propose FedLLM-Bench, which involves 8 training methods, 4 training datasets, and 6 evaluation metrics, to offer a comprehensive testbed for the FedLLM community. FedLLM-Bench encompasses three datasets (e.g., user-annotated multilingual dataset) for federated instruction tuning and one dataset (e.g., user-annotated preference dataset) for federated preference alignment, whose scale of client number ranges from 38 to 747. Our datasets incorporate several representative diversities: language, quality, quantity, instruction, length, embedding, and preference, capturing properties in real-world scenarios. Based on FedLLM-Bench, we conduct experiments on all datasets to benchmark existing FL methods and provide empirical insights (e.g., multilingual collaboration). We believe that our FedLLM-Bench can benefit the FedLLM community by reducing required efforts, providing a practical testbed, and promoting fair comparisons. Code and datasets are available at https://github.com/rui-ye/FedLLM-Bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑