Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications.
For the first time, this work investigates federated NLP in the few-shot scenario (FedFSL). By retrofitting algorithmic advances of pseudo labeling and prompt learning, we first establish a training pipeline that delivers competitive accuracy when only 0.05% (fewer than 100) of the training data is labeled and the remaining is unlabeled. To instantiate the workflow, we further present a system FeS, addressing the high execution cost with novel designs. (1) Curriculum pacing, which injects pseudo labels to the training workflow at a rate commensurate to the learning progress; (2) Representational diversity, a mechanism for selecting the most learnable data, only for which pseudo labels will be generated; (3) Co-planning of a model's training depth and layer capacity. Together, these designs reduce the training delay, client energy, and network traffic by up to 46.0$\times$, 41.2$\times$ and 3000.0$\times$, respectively. Through algorithm/system co-design, FFNLP demonstrates that FL can apply to challenging settings where most training samples are unlabeled.
FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing
A Secure and Efficient Federated Learning Framework for NLP
University of Connecticut(康涅狄格大学);Duke University(杜克大学);Facebook(脸书);Microsoft(微软);Northeastern University(东北大学);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks
School of Automation, Guangdong University of Technology(广东工业大学自动化学院);School of Electronics Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);School of Software Engineering, Beijing Jiaotong University(北京交通大学软件学院)
Towards Energy Efficient Federated Learning over 5G+ Mobile Devices
University of Houston(休斯顿大学);Xidian University(西安电子科技大学);University of Florida(佛罗里达大学)
机器学习算法、5G及5G+无线通信和人工智能硬件实现的不断融合催生了5G+移动设备上的联邦学习(FL),将人工智能功能推向移动设备,开启了设备上人工智能应用的新时代。尽管FL取得了显着进展,但巨大的能耗是限制FL在电池受限的5G+移动设备上发展的最重要障碍之一。为了解决这个问题,在本文中,我们研究了如何通过在“工作”的能耗(即,本地计算)和用于“通话”的(即,无线通信),以提高整体能源效率。具体来说,我们首先研究图形处理单元(GPU)计算和无线传输的能耗模型。然后,我们概述了将FL过程与节能学习技术(例如,梯度稀疏化、权重量化、修剪等)。最后,我们从能源效率的角度提出了FL over 5G+移动设备的几个潜在的未来研究方向。
Auction Based Clustered Federated Learning in Mobile Edge Computing System
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);Cyberspace Security Research Center, Peng Cheng Laboratory(鹏城实验室网络空间安全研究中心);School of Electrical and Data Engineering, University of Technology Sydney(悉尼科技大学电气与数据工程学院);Department of Computer Science and Technology, Fuzhou University(福州大学计算机科学与技术学院);Department of Computer Science, Electrical and Space Engineering, Lulea University of Technology(吕勒奥理工大学计算机、电气与空间工程系)
Federated Learning with Heterogeneous Labels and Models for Mobile Activity Monitoring
Global AI Accelerator, Ericsson(爱立信全球人工智能加速器);Ericsson Research(爱立信研究院)
各种医疗保健应用,如辅助生活、跌倒检测等,需要通过人类活动识别(HAR)对用户行为进行建模。这样的应用需要使用机器学习技术来表征来自多个资源受限的用户设备的见解,以用于有效的个性化活动监视。基于设备的联邦学习被证明是分布式协作机器学习的一种有效方法。然而,在解决用户之间的统计(非IID数据)和模型异质性方面存在各种挑战。此外,在本文中,我们探索了一个新的挑战-在联邦学习过程中处理用户之间标签(活动)的异质性。为此,我们提出了一个框架,联邦标签为基础的聚合,利用重叠的信息增益跨活动使用模型蒸馏更新。我们还建议,模型分数的联邦传输是足够的,而不是从设备到服务器的模型权重传输。在Raspberry Pi 2上使用Heteroimmune Human Activity Recognition(HHAR)数据集(有四个活动用于有效阐明结果)进行的实证评估表明,平均确定性准确度至少提高了~ 11.01%,从而证明了我们提出的框架的设备上功能。
A Practical Cross-Device Federated Learning Framework over 5G Networks
North China Electric Power University(华北电力大学);Fayetteville State University(费耶特维尔州立大学);Stevens Institute of Technology(史蒂文斯理工学院);Qatar University(卡塔尔大学)
When Crowdsensing Meets Federated Learning: Privacy-Preserving Mobile Crowdsensing System
School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院);College of Mathematics and Computer Science, Fuzhou University(福州大学数学与计算机科学学院);Peng Cheng Laboratory(鹏城实验室);School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院)
Energy Efficient Federated Learning over Heterogeneous Mobile Devices via Joint Design of Weight Quantization and Wireless Transmission
University of Houston(休斯顿大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of Science and Technology of China(中国科学技术大学);University of Florida(佛罗里达大学)
Automated Federated Learning in Mobile Edge Networks -- Fast Adaptation and Convergence
Singapore University of Design and Technology(新加坡科技设计大学);East China Normal University(华东师范大学);University of Electronic Science and Technology of China(电子科技大学);Beihang University(北京航空航天大学)
Semi-Synchronous Personalized Federated Learning over Mobile Edge Networks
Singapore University of Design and Technology(新加坡科技设计大学);Shenzhen University(深圳大学);East China Normal University(华东师范大学);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Federated Learning in Mobile Networks: A Comprehensive Case Study on Traffic Forecasting
Democritus University of Thrace(德谟克利特色雷斯大学);Imperial College London(帝国理工学院);Centre Tecnològic de Telecomunicacions de Catalunya (CTTC/CERCA)(加泰罗尼亚电信技术中心(CTTC/CERCA));Nokia Bell Labs(诺基亚贝尔实验室);IMT Atlantique(大西洋电信学校);Inria(法国国家信息与自动化研究所);LS2N(南特计算机科学与实验室(注:此处LS2N为实验室,按要求保留独立机构名))
Benchmarking Differential Privacy and Federated Learning for BERT Models
Manipal Institute of Technology(马尼帕尔理工学院);Carnegie Mellon University(卡内基梅隆大学);OpenMined;Yildiz Technical University(耶尔德兹技术大学);Ford Motor Company(福特汽车公司);University of California, San Diego(加州大学圣地亚哥分校)
REWAFL: Residual Energy and Wireless Aware Participant Selection for Efficient Federated Learning over Mobile Devices
Beijing University of Posts and Telecommunications(北京邮电大学);Peng Cheng Laboratory(鹏城实验室);University of Houston(休斯顿大学);University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)
FedQAS: Privacy-aware machine reading comprehension with federated learning
Uppsala University, Sweden(瑞典乌普萨拉大学);Scaleout Systems, Sweden(瑞典Scaleout Systems公司);Department of Information Technology(信息技术系);Division of Scientific Computing(科学计算部)
SDBA: A Stealthy and Long-Lasting Durable Backdoor Attack in Federated Learning
Department of Information and Communication Engineering, Major in Information Security, Chosun University(信息与通信工程系,信息安全专业,全州大学);AI Data Security Research Section, Electronics and Telecommunications Research Institute(人工智能数据安全研究部,电子电信研究院);Department of Convergence Science, Kongju National University(融合科学系, Kongju国立大学)
FedMood: Federated Learning on Mobile Health Data for Mood Detection
School of Cyber Science and Technology, Beihang University(北京航空航天大学网络空间科学与技术学院);Beijing Advanced Innovation Center for Big Data and Brain Computing(北京大数据与脑计算高精尖创新中心);University of Illinois at Chicago(伊利诺伊大学芝加哥分校);Fordham University(福特汉姆大学);Lehigh University(理海大学)