arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.12400cs.LGcs.DCeess.SP
收藏

通信高效的联邦线性与深度广义典型相关分析

Communication-Efficient Federated Linear and Deep Generalized Canonical Correlation Analysis

Oregon State University(俄勒冈州立大学)

Sagar Shrestha, Xiao Fu

AI总结:

本文针对联邦学习设置下GCCA通信开销过高的问题,提出一种基于最大方差的通信高效联邦线性和深度GCCA框架,通过量化交换信息大幅降低开销,并在几乎不损失精度下实现次线性收敛。

摘要:

经典和深度广义典型相关分析(GCCA)算法分别通过线性变换和神经网络,从多个“视图”(例如音频和图像)中寻找数据实体的低维公共表示。当视图在不同的计算代理(例如组织和边缘设备)处获取和存储,且由于隐私或通信成本的考虑而不希望共享数据时,基于联邦学习的GCCA具有充分的动机。在联邦学习中,视图保存在代理本地,仅允许与中央服务器进行派生的、有限的信息交换。然而,将现有的GCCA算法应用于此类联邦学习设置可能会产生过高的通信开销。本文在最大方差(MAX-VAR)公式下,为线性和深度GCCA提出了一种通信高效的联邦学习框架。通过积极压缩(通过量化)计算代理与中央控制器之间的交换信息来解决开销问题。与未量化版本相比,我们的实证研究表明,所提出的算法在通信开销上大幅减少,且几乎不损失准确性和收敛速度。本文还给出了严格的收敛性分析,这是一项不平凡的工作。通用的联邦优化结果并未涵盖GCCA的特殊问题结构。我们的结果表明,即使在重量化和随机近似下,针对线性和深度GCCA的算法也能以次线性速率收敛到临界点。此外,在线性MAX-VAR情况下,量化算法在合理条件下以几何速率收敛到全局最优解。合成和真实数据实验展示了所提方法的有效性。

英文摘要:

Classic and deep generalized canonical correlation analysis (GCCA) algorithms seek low-dimensional common representations of data entities from multiple ``views'' (e.g., audio and image) using linear transformations and neural networks, respectively. When the views are acquired and stored at different computing agents (e.g., organizations and edge devices) and data sharing is undesired due to privacy or communication cost considerations, federated learning-based GCCA is well-motivated. In federated learning, the views are kept locally at the agents and only derived, limited information exchange with a central server is allowed. However, applying existing GCCA algorithms onto such federated learning settings may incur prohibitively high communication overhead. This work puts forth a communication-efficient federated learning framework for both linear and deep GCCA under the maximum variance (MAX-VAR) formulation. The overhead issue is addressed by aggressively compressing (via quantization) the exchanging information between the computing agents and a central controller. Compared to the unquantized version, our empirical study shows that the proposed algorithm enjoys a substantial reduction of communication overheads with virtually no loss in accuracy and convergence speed. Rigorous convergence analyses are also presented, which is a nontrivial effort. Generic federated optimization results do not cover the special problem structure of GCCA. Our result shows that the proposed algorithms for both linear and deep GCCA converge to critical points at a sublinear rate, even under heavy quantization and stochastic approximations. In addition, in the linear MAX-VAR case, the quantized algorithm approaches a global optimum in a geometric rate under reasonable conditions. Synthetic and real-data experiments are used to showcase the effectiveness of the proposed approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑