arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.11204cs.LGcs.AIcs.DC
收藏

通过合成中间标签进行分区神经网络训练

Partitioned Neural Network Training via Synthetic Intermediate Labels

Yeditepe University(耶迪特佩大学)

Cevat Volkan Karadağ, Nezih Topaloğlu

AI总结:

本文提出通过将神经网络分区到多个GPU并利用随机生成的合成中间标签训练各分段,以降低内存和通信开销,实验表明该方法在扩展MNIST上可保持与传统训练相近的准确率。

摘要:

大规模神经网络架构,尤其是深度学习模型的迅速增多,给资源密集型训练带来了挑战。GPU内存限制已成为训练此类大型模型的显著瓶颈。现有策略,包括数据并行、模型并行、流水线并行和全分片数据并行,只能提供部分解决方案。其中,模型并行能够将整个模型分布到多个GPU上,但这些分区之间的数据通信会降低训练速度。此外,在每个GPU上存储辅助参数所需的大量内存开销进一步增加了计算需求。本研究不使用整个模型进行训练,而是主张将模型在多个GPU上进行分区,并生成合成中间标签来训练各个分段。这些标签通过随机过程产生,能够减轻内存开销和计算负载。该方法实现了更高效的训练过程,在保持模型准确性的同时最大限度地减少了数据通信。为验证该方法,将一个6层全连接神经网络分为两部分,并在扩展MNIST数据集上评估其性能。实验结果表明,所提出的方法达到了与传统训练方法相似的测试准确率,同时显著降低了内存和计算需求。这项工作有助于缓解训练大型神经网络资源密集的问题,为更高效的深度学习模型开发铺平了道路。

英文摘要:

The proliferation of extensive neural network architectures, particularly deep learning models, presents a challenge in terms of resource-intensive training. GPU memory constraints have become a notable bottleneck in training such sizable models. Existing strategies, including data parallelism, model parallelism, pipeline parallelism, and fully sharded data parallelism, offer partial solutions. Model parallelism, in particular, enables the distribution of the entire model across multiple GPUs, yet the ensuing data communication between these partitions slows down training. Additionally, the substantial memory overhead required to store auxiliary parameters on each GPU compounds computational demands. Instead of using the entire model for training, this study advocates partitioning the model across GPUs and generating synthetic intermediate labels to train individual segments. These labels, produced through a random process, mitigate memory overhead and computational load. This approach results in a more efficient training process that minimizes data communication while maintaining model accuracy. To validate this method, a 6-layer fully connected neural network is partitioned into two parts and its performance is assessed on the extended MNIST dataset. Experimental results indicate that the proposed approach achieves similar testing accuracies to conventional training methods, while significantly reducing memory and computational requirements. This work contributes to mitigating the resource-intensive nature of training large neural networks, paving the way for more efficient deep learning model development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑