矩阵计算是边缘计算系统的基本构建模块，由于其在 AI/ML 训练和推理过程中的应用，近期需求大幅上升。现有的分布式矩阵计算方法将子矩阵的编码组合分配给工作节点，以建立对较慢节点（称为掉队者）的弹性。然而，在边缘学习场景中，这些方法会损害边缘服务器上原始矩阵通常具有的稀疏性。在本研究中，我们考虑增强此类方法的挑战，以便在跨边缘设备分配任务时保持输入稀疏性，从而保留相关的计算效率提升。首先，我们给出了编码权重的下界，即为获得编码子矩阵而需要组合的子矩阵数量，以便在给定设备数量及其存储约束下，对可能的最大数量掉队设备提供弹性。接下来，我们提出了恰好满足编码权重下界的分布式矩阵计算方案。在 Amazon Web Services（AWS）中进行的数值实验验证了我们关于稀疏矩阵掉队者缓解和计算速度的论断。
英文摘要：
Matrix computations are a fundamental building-block of edge computing systems, with a major recent uptick in demand due to their use in AI/ML training and inference procedures. Existing approaches for distributing matrix computations involve allocating coded combinations of submatrices to worker nodes, to build resilience to slower nodes, called stragglers. In the edge learning context, however, these approaches will compromise sparsity properties that are often present in the original matrices found at the edge server. In this study, we consider the challenge of augmenting such approaches to preserve input sparsity when distributing the task across edge devices, thereby retaining the associated computational efficiency enhancements. First, we find a lower bound on the weight of coding, i.e., the number of submatrices to be combined to obtain coded submatrices, to provide the resilience to the maximum possible number of straggler devices (for given number of devices and their storage constraints). Next we propose distributed matrix computation schemes which meet the exact lower bound on the weight of the coding. Numerical experiments conducted in Amazon Web Services (AWS) validate our assertions regarding straggler mitigation and computation speed for sparse matrices.
Flexible Coded Distributed Convolution Computing for Enhanced Straggler Resilience and Numerical Stability in Distributed CNNs
College of Computer Science, Chongqing University（重庆大学计算机学院）；School of Electrical Engineering, Korea University（韩国大学电气工程学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；City University of Hong Kong（香港城市大学）
Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
School of Computing Informatics and Decision Systems Engineering at Arizona State University（亚利桑那州立大学计算、信息科学与工程学院）；Computer Science and Artificial Intelligence Laboratory at Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）；School of Computer Science at University of California, Los Angeles（加利福尼亚大学洛杉矶分校计算机科学学院）；Parallel Computing Lab at Intel Labs（英特尔实验室并行计算实验室）
