While deep neural net inference was considered a task for servers only, latest advances in technology allow the task of inference to be moved to mobile and embedded devices, desired for various reasons ranging from latency to privacy. These devices are not only limited by their compute power and battery, but also by their inferior physical memory and cache, and thus, an efficient memory manager becomes a crucial component for deep neural net inference at the edge. We explore various strategies to smartly share memory buffers among intermediate tensors in deep neural nets. Employing these can result in up to 11% smaller memory footprint than the state of the art.
vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs
School of Computer Science, Peking University（北京大学计算机学院）；School of Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程学院）；OctoAI（OctoAI公司）
PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
Tencent Inc（腾讯公司）；National University of Singapore（新加坡国立大学）
预训练模型（PTM）正在彻底改变人工智能（AI）技术。然而，PTM训练的硬件要求非常高，使其成为一小部分人的游戏。因此，我们提出了PatrickStar系统，以降低PTM的硬件要求，并使其可供任何人使用。PatrickStar使用CPU-GPU异构内存空间存储模型数据。与现有的工作不同，我们组织的模型数据在内存块和动态分布在异构内存。在预热迭代中收集的运行时内存统计数据的指导下，在异构内存中高效地编排块，并产生更低的CPU-GPU数据传输量和更高的带宽利用率。PatrickStar与零冗余优化器共生，可扩展到多个节点上的多个GPU。%使用数据并行。该系统可以在更大的模型和更大的批量上进行训练任务，这是现有工作无法完成的。实验结果表明，PatrickStar将模型规模扩展到DeepSpeed的2.27和2.5倍，并始终表现出显着更高的执行速度。PatricStar还在32 GPU集群上成功运行了175 B GPT 3训练任务。我们的代码可在https://github.com/Tencent/PatrickStar上公开获取。
SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks
Brown University（布朗大学）；Univ. of Electr. Sci. & Tech. of China（电子科技大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）；Pacific Northwest National Laboratory（太平洋西北国家实验室）；Massachusetts Institute of Technology（麻省理工学院）
Risk Analysis in Customer Relationship Management via Quantile Region Convolutional Neural Network-Long Short-Term Memory and Cross-Attention Mechanism
Longyan University（龙岩学院）；National Central University（中央大学）；University of Portsmouth（朴茨茅斯大学）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；Northeastern University（东北大学）
