面向语音转文本任务的通信高效个性化联邦学习
Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks
AI总结：
针对联邦语音转文本任务中通信开销大和数据异质性问题，提出FedLoRA与FedMem的个性化框架，显著降低通信开销并提升个性化性能。
摘要：
为了保护隐私并满足法律法规要求，联邦学习（FL）在训练语音转文本（S2T）系统（包括自动语音识别（ASR）和语音翻译（ST））方面受到了广泛关注。然而，在S2T任务中常用的联邦学习方法（即FedAvg）通常面临两大问题：一是基于整个模型的多轮交互导致的大量通信开销，二是客户端间数据异质性导致的性能下降。为解决这些问题，我们提出了一种个性化联邦S2T框架，该框架引入了FedLoRA——一种轻量级LoRA模块，用于客户端侧调优并与服务器交互，以最小化通信开销；以及FedMem——一个配备k近邻（kNN）分类器的全局模型，该分类器能够捕获客户端特定的分布偏移，从而实现个性化并克服数据异质性。基于Conformer和Whisper骨干模型在CoVoST和GigaSpeech基准上的大量实验表明，我们的方法显著降低了所有S2T任务上的通信开销，并有效地个性化全局模型以克服数据异质性。
英文摘要：
To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems, including automatic speech recognition (ASR) and speech translation (ST). However, the commonly used FL approach (i.e., \textsc{FedAvg}) in S2T tasks typically suffers from extensive communication overhead due to multi-round interactions based on the whole model and performance degradation caused by data heterogeneity among clients.To address these issues, we propose a personalized federated S2T framework that introduces \textsc{FedLoRA}, a lightweight LoRA module for client-side tuning and interaction with the server to minimize communication overhead, and \textsc{FedMem}, a global model equipped with a $k$-nearest-neighbor ($k$NN) classifier that captures client-specific distributional shifts to achieve personalization and overcome data heterogeneity. Extensive experiments based on Conformer and Whisper backbone models on CoVoST and GigaSpeech benchmarks show that our approach significantly reduces the communication overhead on all S2T tasks and effectively personalizes the global model to overcome data heterogeneity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
