We study neural networks whose only non-linear components are multipliers, to test a new training rule in a context where the precise representation of data is paramount. These networks are challenged to discover the rules of matrix multiplication, given many examples. By limiting the number of multipliers, the network is forced to discover the Strassen multiplication rules. This is the mathematical equivalent of finding low rank decompositions of the $n\times n$ matrix multiplication tensor, $M_n$. We train these networks with the conservative learning rule, which makes minimal changes to the weights so as to give the correct output for each input at the time the input-output pair is received. Conservative learning needs a few thousand examples to find the rank 7 decomposition of $M_2$, and $10^5$ for the rank 23 decomposition of $M_3$ (the lowest known). High precision is critical, especially for $M_3$, to discriminate between true decompositions and "border approximations".
Strong Scaling of Matrix Multiplication Algorithms and Memory-Independent Communication Lower Bounds
如果并行算法在P个处理机上的运行时间是1/P线性的,包括所有的通信开销,则该并行算法具有完美的强标度性。具有完美强标度的矩阵乘法的分布式内存并行算法最近才被发现。一种是基于经典矩阵乘法(Solomonik and Demmel,2011),另一种是基于Strassen的快速矩阵乘法(Ballard,Demmel,Holtz,Lipshitz,and Schwartz,2012)。这两种算法都可以完美地扩展,但只能扩展到处理器间通信不再扩展的处理器数量。我们得到了一个内存独立的通信成本的经典和基于Strassen的分布式存储矩阵乘法算法的下界。这些界限意味着,没有经典的或基于Strassen的并行矩阵乘法算法可以强大的规模完全超出上述两个并行算法已经达到的范围。记忆无关的边界和强缩放边界推广到其他算法。
Fast Matrix Multiplication in fp8: Certified Coefficient Optimization and Measured Error
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC)(北京通明湖信息技术应用创新中心);Fudan University Institute of Systems for Advanced Computing(复旦大学先进计算系统研究院);Harbin Institute of Technology(哈尔滨工业大学);Key Lab of HCST (PKU), MOE(北京大学高可信软件技术教育部重点实验室;北京大学计算机学院);SCS, Peking University(上海开放计算系统研究院);Shanghai Institute of Systems for Open Computing
University of California, Berkeley(加州大学伯克利分校);NVIDIA Corp.(英伟达公司);University of Colorado Denver(科罗拉多大学丹佛分校);Lawrence Berkeley National Lab(劳伦斯伯克利国家实验室);Microsoft Corp.(微软公司)