Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps. However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy. We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity. We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality. For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%. K-matrices can also simplify hand-engineered pipelines -- we replace filter bank feature computation in speech data preprocessing with a learnable kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task. In addition, K-matrices can capture latent structure in models: for a challenging permuted image classification task, a K-matrix based representation of permutations is able to learn the right latent structure and improves accuracy of a downstream convolutional model by over 9%. We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task.
SKOLR: Structured Koopman Operator Linear RNN for Time-Series Forecasting
Department of Electrical and Computer Engineering, McGill University, Montreal, Canada（麦吉尔大学电气与计算机工程系）；Amazon Science. This work does not relate to the author’s position at Amazon（亚马逊科学）；Mila - Quebec Artificial Intelligence Institute, Montreal, Canada（魁北克人工智能研究所）；ILLS - International Laboratory on Learning Systems, Montreal, Canada（学习系统国际实验室）
Structured Latent Space Modeling over Multi-Scale Temporal Patches for Multivariate Time Series Forecasting
School of Computing and Data Science, The University of Hong Kong（香港大学计算与数据科学学院）；Innovation Engineering College, Macau University of Science and Technology（澳门科技大学创新工程学院）
Department of Computer Science \& Engineering Hong Kong University of Science；Faculty of Engineering \& Information Technology University of Technology Sydney Sydney, New South Wales, Australia
