DIMAT：面向深度学习模型的去中心化迭代合并与训练
DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models
AI总结：
针对去中心化深度学习模型更新通信与计算开销高的问题，提出DIMAT范式，结合本地训练与模型合并技术，实现最优收敛速率与更紧误差界，在多视觉任务上验证了其高效性与适应性。
摘要：
去中心化深度学习算法的最新进展已在基于大型预训练模型的各类任务上展现出顶尖性能。然而，要达到这种竞争力水平的一个关键前提是更新这些模型时需付出巨大的通信与计算开销，这阻碍了它们在实际场景中的应用。为解决这一问题，我们从无需额外训练的先进模型合并技术中汲取灵感，提出了去中心化迭代合并与训练（Decentralized Iterative Merging-And-Training, DIMAT）范式——一种新型的去中心化深度学习框架。在DIMAT中，每个智能体基于自身本地数据进行训练，并使用激活匹配等先进模型合并技术与相邻智能体定期合并，直至收敛。DIMAT在多种一阶方法下可证明以非凸函数的最优可用速率收敛，同时与现有主流方法相比具有更紧的误差界。我们开展了全面的实证分析，以验证DIMAT在来自多个数据集的各类计算机视觉任务上优于基线方法。实证结果验证了我们的理论结论：在独立同分布（IID）和非IID数据下，DIMAT能更快获得更高的初始准确率提升，且产生的通信开销更低。该DIMAT范式为未来的去中心化学习提供了新的机遇，通过稀疏轻量的通信与计算提升了其对现实世界的适应性。
英文摘要：
Recent advances in decentralized deep learning algorithms have demonstrated cutting-edge performance on various tasks with large pre-trained models. However, a pivotal prerequisite for achieving this level of competitiveness is the significant communication and computation overheads when updating these models, which prohibits the applications of them to real-world scenarios. To address this issue, drawing inspiration from advanced model merging techniques without requiring additional training, we introduce the Decentralized Iterative Merging-And-Training (DIMAT) paradigm--a novel decentralized deep learning framework. Within DIMAT, each agent is trained on their local data and periodically merged with their neighboring agents using advanced model merging techniques like activation matching until convergence is achieved. DIMAT provably converges with the best available rate for nonconvex functions with various first-order methods, while yielding tighter error bounds compared to the popular existing approaches. We conduct a comprehensive empirical analysis to validate DIMAT's superiority over baselines across diverse computer vision tasks sourced from multiple datasets. Empirical results validate our theoretical claims by showing that DIMAT attains faster and higher initial gain in accuracy with independent and identically distributed (IID) and non-IID data, incurring lower communication overhead. This DIMAT paradigm presents a new opportunity for the future decentralized learning, enhancing its adaptability to real-world with sparse and light-weight communication and computation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
