arXiv 2505.018512025-05-03cs.CV

缓解联邦视觉语言模型中的群体级公平性差异

Mitigating Group-Level Fairness Disparities in Federated Visual Language Models

Great Bay University（大湾大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shenzhen University（深圳大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）

Chaomeng Chen, Zitong Yu, Junhao Dong, Sen Su, Linlin Shen, Shutao Xia, Xiaochun Cao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出联邦公平提示微调框架FVL-FP，通过跨层公平提示、子空间正交投影和公平感知融合三个组件，在保持性能的同时将群体公平性差异平均降低45%。

摘要：

视觉语言模型（VLMs）在多模态任务中展现了卓越的能力，但在跨人口群体维持公平性方面面临挑战，尤其是在联邦学习（FL）环境中部署时。本文通过引入FVL-FP来解决联邦视觉语言模型中的群体公平性这一关键问题，FVL-FP是一个将联邦学习与公平提示微调技术相结合的新型框架。我们专注于在保持模型性能的同时缓解人口统计学偏见，通过三个创新组件实现：（1）跨层人口统计学公平提示（CDFP），通过反事实正则化调整可能存在偏见的嵌入；（2）人口统计学子空间正交投影（DSOP），通过将公平提示文本映射到群体子空间来消除图像表示中的人口统计学偏见；（3）公平感知提示融合（FPF），基于性能和公平性指标动态平衡客户端贡献。在四个基准数据集上的广泛评估表明，与标准联邦学习方法相比，我们的方法将人口统计学差异平均减少了45%，同时将任务性能保持在最先进结果的6%以内。FVL-FP有效解决了联邦设置中非独立同分布（non-IID）数据分布的挑战，并在提供显著公平性收益的同时引入了最小的计算开销。我们的工作为在隐私保护的多模态系统中确保跨人口群体公平性能这一关键挑战提供了一种参数高效的解决方案。

英文摘要：

Visual language models (VLMs) have shown remarkable capabilities in multimodal tasks but face challenges in maintaining fairness across demographic groups, particularly when deployed in federated learning (FL) environments. This paper addresses the critical issue of group fairness in federated VLMs by introducing FVL-FP, a novel framework that combines FL with fair prompt tuning techniques. We focus on mitigating demographic biases while preserving model performance through three innovative components: (1) Cross-Layer Demographic Fair Prompting (CDFP), which adjusts potentially biased embeddings through counterfactual regularization; (2) Demographic Subspace Orthogonal Projection (DSOP), which removes demographic bias in image representations by mapping fair prompt text to group subspaces; and (3) Fair-aware Prompt Fusion (FPF), which dynamically balances client contributions based on both performance and fairness metrics. Extensive evaluations across four benchmark datasets demonstrate that our approach reduces demographic disparity by an average of 45\% compared to standard FL approaches, while maintaining task performance within 6\% of state-of-the-art results. FVL-FP effectively addresses the challenges of non-IID data distributions in federated settings and introduces minimal computational overhead while providing significant fairness benefits. Our work presents a parameter-efficient solution to the critical challenge of ensuring equitable performance across demographic groups in privacy-preserving multimodal systems.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。