基于适配器的多语言神经机器翻译通信高效联邦学习
Communication Efficient Federated Learning for Multilingual Neural Machine Translation with Adapter
AI总结：
提出通信高效的Fed-MNMT框架，通过冻结PLMs并仅传输轻量级适配器模块，结合聚类策略解决语言分布差异导致的参数冲突，在保持或提升性能的同时将通信成本降低98%以上。
摘要：
联邦多语言神经机器翻译（Fed-MNMT）已成为语言资源有限的机构的一种有前途的范式。这种方法允许多个机构作为客户端，通过模型同步来训练统一模型，而不是收集敏感数据进行集中训练。这显著降低了语料库收集成本并保护了数据隐私。然而，随着预训练语言模型（PLMs）规模不断增大，同步过程中传输参数的通信成本已成为训练速度的瓶颈。在本文中，我们提出了一种通信高效的Fed-MNMT框架，通过保持PLMs冻结并仅在客户端之间传输轻量级适配器模块来解决此问题。由于不同语言对在数据分布上表现出显著差异，客户端的适配器参数可能相互冲突。为解决此问题，我们探索了各种聚类策略，对参数进行分组以进行整合，并减轻冲突参数的负面影响。实验结果表明，与竞争性基线相比，我们的框架在实现相似甚至更好性能的同时，将通信成本降低了98%以上。进一步分析表明，聚类策略有效解决了语言差异问题，而修剪适配器模块进一步提高了通信效率。
英文摘要：
Federated Multilingual Neural Machine Translation (Fed-MNMT) has emerged as a promising paradigm for institutions with limited language resources. This approach allows multiple institutions to act as clients and train a unified model through model synchronization, rather than collecting sensitive data for centralized training. This significantly reduces the cost of corpus collection and preserves data privacy. However, as pre-trained language models (PLMs) continue to increase in size, the communication cost for transmitting parameters during synchronization has become a training speed bottleneck. In this paper, we propose a communication-efficient Fed-MNMT framework that addresses this issue by keeping PLMs frozen and only transferring lightweight adapter modules between clients. Since different language pairs exhibit substantial discrepancies in data distributions, adapter parameters of clients may conflict with each other. To tackle this, we explore various clustering strategies to group parameters for integration and mitigate the negative effects of conflicting parameters. Experimental results demonstrate that our framework reduces communication cost by over 98% while achieving similar or even better performance compared to competitive baselines. Further analysis reveals that clustering strategies effectively solve the problem of linguistic discrepancy and pruning adapter modules further improves communication efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
