Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)
Ericsson AB（爱立信公司）；KTH Royal Institute of Technology（KTH皇家理工学院）；RISE Research Institutes of Sweden（瑞典RISE研究院）；AI Sweden（瑞典人工智能研究院）；Chalmers University of Technology（查尔姆斯理工大学）
Federated Learning (FL) is a promising framework for distributed learning when data is private and sensitive. However, the state-of-the-art solutions in this framework are not optimal when data is heterogeneous and non-Independent and Identically Distributed (non-IID). We propose a practical and robust approach to personalization in FL that adjusts to heterogeneous and non-IID data by balancing exploration and exploitation of several global models. To achieve our aim of personalization, we use a Mixture of Experts (MoE) that learns to group clients that are similar to each other, while using t…
HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning
Department of ECE and INMC, Seoul National University, Seoul, South Korea（首尔国立大学电子与信息科学系及网络信息中心，首尔，韩国）；Graduate School of Data Science, Seoul National University, Seoul, South Korea（首尔国立大学数据科学研究生院，首尔，韩国）；Department of Computer Science, University of Colorado Boulder, CO, USA（科罗拉多大学博尔德分校计算机科学系，科罗拉多州，美国）
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）
Federated learning faces huge challenges from model overfitting due to the lack of data and statistical diversity among clients. To address these challenges, this paper proposes a novel personalized federated learning method via Bayesian variational inference named pFedBayes. To alleviate the overfitting, weight uncertainty is introduced to neural networks for clients and the server. To achieve personalization, each client updates its local distribution parameters by balancing its construction error over private data and its KL divergence with global distribution from the server. Theoretical a…
City University of Hong Kong（香港城市大学）；Hong Kong Metropolitan University（香港 Metropolitan 大学）；Xi’an Jiaotong University（西安交通大学）；Southeast University（东南大学）；Jinan University（吉林大学）；CityU Shenzhen Research Institute（城大深圳研究院）
Mohammed VI Polytechnic University（穆罕默德六世理工大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Southern Denmark（南丹麦大学）
In contrast to training traditional machine learning (ML) models in data centers, federated learning (FL) trains ML models over local datasets contained on resource-constrained heterogeneous edge devices. Existing FL algorithms aim to learn a single global model for all participating devices, which may not be helpful to all devices participating in the training due to the heterogeneity of the data across the devices. Recently, Hanzely and Richtárik (2020) proposed a new formulation for training personalized FL models aimed at balancing the trade-off between the traditional global model and the…
Enhancing Convergence, Privacy and Fairness for Wireless Personalized Federated Learning: Quantization-Assisted Min-Max Fair Scheduling
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（信息与通信工程学院，北京邮电大学）；School of Computing, Macquarie University（计算学院，麦考瑞大学）；Department of Broadband Communication, Peng Cheng Laboratory（宽带通信部，鹏城实验室）；China Telecom, Sichuan Branch（中国电信四川分公司）
FedeCouple: Fine-Grained Balancing of Global-Generalization and Local-Adaptability in Federated Learning
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（计算能力网络与信息安全部分省计算机科学中心，青岛科技大学（山东省科学院））；Shandong Provincial Key Laboratory of Industrial Network and Information System Security, Shandong Fundamental Research Center for Computer Science（山东省工业网络与信息系统安全省重点实验室，山东省计算机科学基础研究中心）；School of Computer Science and Technology, Shandong University（山东大学计算机科学与技术学院）；School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院）；State Key Lab. of Industrial Control Technology, Zhejiang University（浙江省工业控制技术重点实验室，浙江大学）
