We study rectangular matrix multiplication in the low-bandwidth model of distributed computing. There are $n$ computers; initially the input matrices are distributed evenly between computers, and in each communication round every computer can send and receive an $O(\log n)$-bit message. Eventually each computer must output its designated part of the product matrix.
While prior work has focused primarily on square $n \times n$ multiplication under various sparsity assumptions, we study rectangular instances with no sparsity assumption. We denote by $\langle a,b,c\rangle$ the task of multiplying an $a\times b$ matrix by a $b\times c$ matrix in this model. We concentrate on two natural aspect ratios, $\langle n,d,n\rangle$ and $\langle d,n,d\rangle$, for $d \le n$, and we study how the round complexity depends on $n$ and $d$.
When $d \to n$, both $\langle n,d,n\rangle$ and $\langle d,n,d\rangle$ approach $\langle n,n,n\rangle$, which is the usual task of multiplying square matrices. If we consider multiplication over semirings, the current best upper bound in that case is $O(n^{4/3})$ rounds, and there is a trivial unconditional lower bound of $Ω(n)$.
We show that for $\langle d,n,d\rangle$, we can achieve the complexity of $\tilde O(d^{4/3})$, which seems like a natural generalization of the upper bound $\tilde O(n^{4/3})$ when $d=n$. However, the case of $\langle n,d,n\rangle$ is fundamentally different, and also exhibits a phase transition. We show that for $d \le \sqrt{n}$, the complexity of $\langle n,d,n\rangle$ is $Θ(d \sqrt{n})$; we have matching upper and lower bounds. However, the behavior is genuinely different in the region $d \ge \sqrt{n}$, where the upper bound is $O(d^{2/3} n^{2/3})$.
Performance Modeling of Streaming Kernels and Sparse Matrix-Vector Multiplication on A64FX
A64 FX CPU是目前Top500排行榜上排名第一的超级计算机。虽然它是一个传统的基于缓存的多核处理器,但其峰值性能和内存带宽可与加速器设备相媲美。为这样一个新的体系结构生成高效的代码需要很好地理解其性能特性。利用这些功能,我们构建的执行缓存存储器(ECM)的性能模型的A64 FX处理器在FX 700超级计算机,并验证它使用流循环。我们还确定架构的特点,并获得优化提示。将ECM模型应用于稀疏矩阵向量乘法(SpMV),我们激发了CRS矩阵存储格式不合适的原因以及SELL-C-sigma格式如何通过适当的代码优化实现SpMV的带宽饱和。
Fast Matrix Multiplications for Lookup Table-Quantized LLMs
Massachusetts Institute of Technology(麻省理工学院);High School of Mathematics Plovdiv(普罗夫迪夫数学中学);Carnegie Mellon University(卡内基梅隆大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);Petuum Inc.(派图公司)
FalconGEMM: Surpassing Hardware Peaks with Lower-Complexity Matrix Multiplication
Tencent, Shenzhen, China(腾讯);The Hong Kong Polytechnic University, Hong Kong(香港理工大学);NVIDIA, Beijing, China(英伟达);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);RIKEN, Tokyo, Japan(理化学研究所);Southern University of Science and Technology, Shenzhen, China(南方科技大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China(中国科学院深圳先进技术研究院)