SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
School of Computer Science, Fudan University(复旦大学计算机科学学院);University of Hong Kong(香港大学);University of Maryland(马里兰大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Massachusetts Institute of Technology(麻省理工学院)
The scalability of large language models (LLMs) in handling high-complexity models and large-scale datasets has led to tremendous successes in pivotal domains. While there is an urgent need to acquire more training data for LLMs, a concerning reality is the depletion of high-quality public datasets within a few years. In view of this, the federated learning (FL) LLM fine-tuning paradigm recently has been proposed to facilitate collaborative LLM fine-tuning on distributed private data, where multiple data owners collaboratively fine-tune a shared LLM without sharing raw data. However, the staggering model size of LLMs imposes heavy computing and communication burdens on clients, posing significant barriers to the democratization of the FL LLM fine-tuning paradigm. To address this issue, split learning (SL) has emerged as a promising solution by offloading the primary training workload to a server via model partitioning while exchanging activation/activation's gradients with smaller data sizes rather than the entire LLM. Unfortunately, research on the SL LLM fine-tuning paradigm is still in its nascent stage. To fill this gap, in this paper, we propose the first SL LLM fine-tuning framework, named SplitLoRA. SplitLoRA is built on the split federated learning (SFL) framework, amalgamating the advantages of parallel training from FL and model splitting from SL and thus greatly enhancing the training efficiency. It is worth noting that SplitLoRA is the inaugural open-source benchmark for SL LLM fine-tuning, providing a foundation for research efforts dedicated to advancing SL LLM fine-tuning. Extensive simulations validate that SplitLoRA achieves target accuracy in significantly less time than state-of-the-art LLM fine-tuning frameworks, demonstrating the superior training performance of SplitLoRA. The project page is available at https://fduinc.github.io/splitlora/.
TSFLora: Token-Compressed Split Fine-Tuning for Wireless Edge Networks
School of Computer Science and Engineering(计算机科学与工程学院);Center for Intelligent Networking and Communications(智能网络与通信中心);School of Computer Science(计算机科学学院);Beijing University of Posts and Telecommunications(北京邮电大学)
HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models
Institute of Space Internet, Fudan University(复旦大学空间互联网研究所);School of Computer Science, Fudan University(复旦大学计算机科学学院);Department of Electrical and Electronic Engineering, University of Hong Kong(香港大学电子与电气工程系);Mohamed bin Zayed University of Artificial Intelligence(莫扎伊德人工智能大学);Massachusetts Institute of Technology(麻省理工学院);Data61, CSIRO(CSIRO Data61);School of Computer Engineering, Nanyang Technological University(南洋理工大学计算机工程学院)
Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networks
School of Electronics and Information Engineering, Harbin Institute of Technology(哈尔滨工业大学电子与信息工程学院);Pengcheng Laboratory(鹏城实验室);School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院)
Decoupled Conformal Optimisation: Efficient Prediction Sets via Independent Tuning and Calibration
Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);UKRI AI Centre for Doctoral Training in Decision Making for Complex Systems(UKRI人工智能博士培训中心(复杂系统决策));Department of Computer Science, Aalto University(艾尔沃斯大学计算机科学系);ELLIS Institute, Finland(芬兰ELLIS研究所)
ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over the Network Edge
School of Informatics, Xiamen University(厦门大学信息学院);National University of Singapore(新加坡国立大学);Teleinfo iF-Labs, China Academy of Information and Communications Technology(Teleinfo iF-Labs,中国信息通信技术研究院);Department of Electrical Engineering, University at Buffalo-SUNY(电气工程系,布法罗大学-纽约州立大学)