arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.03285cs.DC
收藏

基于异步分布式回放缓冲区的高效数据并行持续学习

Efficient Data-Parallel Continual Learning with Asynchronous Distributed Rehearsal Buffers

Thomas Bouvier, Bogdan Nicolae, Hugo Chaugier, Alexandru Costan, Ian Foster, Gabriel Antoniu

AI总结:

提出基于异步分布式回放缓冲区的方法,在多GPU上高效补充数据并行训练,在保持高准确性的同时实现短运行时间和可扩展性。

摘要:

深度学习已成为从大量数据中提取有价值信息的强大方法。然而,当新训练数据持续到达(即从一开始就没有完全可用)时,增量训练会遭受灾难性遗忘(即新模式得到强化,而先前获得的知识则被牺牲)。每次新训练数据可用时从头开始训练将导致极长的训练时间和海量的数据积累。基于回放的持续学习在解决灾难性遗忘挑战方面显示出了前景,但迄今为止的研究尚未解决性能和可扩展性问题。为了填补这一空白,我们提出了一种基于分布式回放缓冲区的方法,该方法有效地补充了多GPU上的数据并行训练,使我们能够在保持高准确性的同时实现短运行时间和可扩展性。它利用一组缓冲区(每个GPU本地)并使用几种异步技术以尴尬并行的方式更新这些本地缓冲区,同时处理使用无偏全局采样来增强输入小批量(馈送给模型的训练样本组)所需的通信开销。在本文中,我们探讨了这种方法对分类模型的益处。我们在ThetaGPU超级计算机的最多128个GPU上进行了大量实验,将我们的方法与代表从头开始训练(准确性上限)和增量训练(下限)的基线进行了比较。结果表明,基于回放的持续学习实现了接近上限的top-5分类准确性,同时表现出接近下限的运行时间。

英文摘要:

Deep learning has emerged as a powerful method for extracting valuable information from large volumes of data. However, when new training data arrives continuously (i.e., is not fully available from the beginning), incremental training suffers from catastrophic forgetting (i.e., new patterns are reinforced at the expense of previously acquired knowledge). Training from scratch each time new training data becomes available would result in extremely long training times and massive data accumulation. Rehearsal-based continual learning has shown promise for addressing the catastrophic forgetting challenge, but research to date has not addressed performance and scalability. To fill this gap, we propose an approach based on a distributed rehearsal buffer that efficiently complements data-parallel training on multiple GPUs, allowing us to achieve short runtime and scalability while retaining high accuracy. It leverages a set of buffers (local to each GPU) and uses several asynchronous techniques for updating these local buffers in an embarrassingly parallel fashion, all while handling the communication overheads necessary to augment input mini-batches (groups of training samples fed to the model) using unbiased, global sampling. In this paper we explore the benefits of this approach for classification models. We run extensive experiments on up to 128 GPUs of the ThetaGPU supercomputer to compare our approach with baselines representative of training-from-scratch (the upper bound in terms of accuracy) and incremental training (the lower bound). Results show that rehearsal-based continual learning achieves a top-5 classification accuracy close to the upper bound, while simultaneously exhibiting a runtime close to the lower bound.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑