Coop:内存不是一种商品
Coop: Memory is not a Commodity
AI总结:
Coop通过滑动窗口连续逐出、低成本张量划分和可重算就地操作,协同优化张量分配与重物化,在八个DNN上实现最高2倍内存节省并显著降低计算开销、搜索延迟和内存碎片化。
摘要:
张量重物化通过在检查点保存模型并按需重算被逐出的张量,使得深度神经网络(DNN)能够在有限的内存预算下进行训练。然而,现有的张量重物化技术忽视了深度学习框架中的内存系统,并隐含地假设不同地址上的空闲内存块是相同的。在这一有缺陷的假设下,不连续张量被逐出,其中一些并未用于分配新张量。这导致了严重的内存碎片化,并增加了潜在重物化的成本。为了解决这一问题,我们提出在滑动窗口内逐出张量,以确保所有逐出都是连续的并立即被使用。此外,我们提出了低成本张量划分和可重算的就地操作,通过优化张量分配进一步降低重物化成本。我们将该方法命名为Coop,因为它是张量分配与张量重物化的协同优化。我们在八个具有代表性的DNN上评估了Coop。实验结果表明,与最先进的基线方法相比,Coop实现了高达2倍的内存节省,并大幅降低了计算开销、搜索延迟和内存碎片化。
英文摘要:
Tensor rematerialization allows the training of deep neural networks (DNNs) under limited memory budgets by checkpointing the models and recomputing the evicted tensors as needed. However, the existing tensor rematerialization techniques overlook the memory system in deep learning frameworks and implicitly assume that free memory blocks at different addresses are identical. Under this flawed assumption, discontiguous tensors are evicted, among which some are not used to allocate the new tensor. This leads to severe memory fragmentation and increases the cost of potential rematerializations. To address this issue, we propose to evict tensors within a sliding window to ensure all evictions are contiguous and are immediately used. Furthermore, we proposed cheap tensor partitioning and recomputable in-place to further reduce the rematerialization cost by optimizing the tensor allocation. We named our method Coop as it is a co-optimization of tensor allocation and tensor rematerialization. We evaluated Coop on eight representative DNNs. The experimental results demonstrate that Coop achieves up to $2\times$ memory saving and hugely reduces compute overhead, search latency, and memory fragmentation compared to the state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。