Tensor network, which originates from quantum physics, is emerging as an efficient tool for classical and quantum machine learning. Nevertheless, there still exists a considerable accuracy gap between tensor network and the sophisticated neural network models for classical machine learning. In this work, we combine the ideas of matrix product state (MPS), the simplest tensor network structure, and residual neural network and propose the residual matrix product state (ResMPS). The ResMPS can be treated as a network where its layers map the "hidden" features to the outputs (e.g., classifications), and the variational parameters of the layers are the functions of the features of the samples (e.g., pixels of images). This is different from neural network, where the layers map feed-forwardly the features to the output. The ResMPS can equip with the non-linear activations and dropout layers, and outperforms the state-of-the-art tensor network models in terms of efficiency, stability, and expression power. Besides, ResMPS is interpretable from the perspective of polynomial expansion, where the factorization and exponential machines naturally emerge. Our work contributes to connecting and hybridizing neural and tensor networks, which is crucial to further enhance our understand of the working mechanisms and improve the performance of both models.
PURe: A Plug-and-Play Product-Unit Residual Module for Vision Networks
Department of Mathematics, Informatics and Technology, University of Applied Sciences Koblenz（科隆应用科学大学数学、信息学与技术系）；Technical University of Munich（慕尼黑技术大学）
Use of operator defect identities in multi-channel signal plus residual-analysis via iterated products and telescoping energy-residuals: Applications to kernels in machine learning
Department of Mathematics, The University of Iowa（数学系，爱荷华大学）；Department of Mathematics and Statistics, Southern Illinois University Edwardsville（数学与统计学系，南伊利诺伊大学爱德华兹维尔分校）；Mathematical Reviews（数学评论）
BandPC: Learning Residual-Band Preconditioner Combinations for Flexible Conjugate Gradient Solvers
Wuhan University of Technology（武汉理工大学）；Embodied Physical Intelligence Center in Civil Engineering, Wuhan University of Technology（武汉理工大学土木工程具身智能中心）；Key Laboratory of Low-Altitude Technology and Smart Urban Renewal, Department of Housing and Urban-Rural Development of Hubei Province, Wuhan University of Technology（湖北省住房和城乡建设厅低空技术与智慧城市更新重点实验室（武汉理工大学））；Sanya Science and Education Innovation Park of Wuhan University of Technology（武汉理工大学三亚科教创新园）
Beijing Institute of Technology（北京理工大学）；Guangdong Laboratory of Machine Perception and Intelligent Computing（广东省机器感知与智能计算实验室）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）
On the Convergence of Gradient Descent on Learning Transformers with Residual Connections
Michigan Institute for Computational Discovery and Engineering, Department of Electrical Engineering and Computer Science, Department of Statistics, University of Michigan（密歇根大学计算发现与工程研究所、电气工程与计算机科学系、统计系）；Department of Computer Science and Engineering, The Ohio State University（俄亥俄州立大学计算机科学与工程系）
