Tensor Networks (TN) offer a powerful framework to efficiently represent very high-dimensional objects. TN have recently shown their potential for machine learning applications and offer a unifying view of common tensor decomposition models such as Tucker, tensor train (TT) and tensor ring (TR). However, identifying the best tensor network structure from data for a given task is challenging. In this work, we leverage the TN formalism to develop a generic and efficient adaptive algorithm to jointly learn the structure and the parameters of a TN from data. Our method is based on a simple greedy approach starting from a rank one tensor and successively identifying the most promising tensor network edges for small rank increments. Our algorithm can adaptively identify TN structures with small number of parameters that effectively optimize any differentiable objective function. Experiments on tensor decomposition, tensor completion and model compression tasks demonstrate the effectiveness of the proposed algorithm. In particular, our method outperforms the state-of-the-art evolutionary topology search [Li and Sun, 2020] for tensor decomposition of images (while being orders of magnitude faster) and finds efficient tensor network structures to compress neural networks outperforming popular TT based approaches [Novikov et al., 2015].
College of Information Science and Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院);Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing(浙江省多模态通信网络与智能信息处理重点实验室);Microsoft Research Asia(微软亚洲研究院)
Structured Adaptive Tensor Prediction for Streaming Data
Michigan Institute for Computational Discovery and Engineering, Department of Statistics, Department of Electrical Engineering and Computer Science, University of Michigan(密歇根大学计算发现与工程研究所、统计系、电气工程与计算机科学系);Department of Statistics, University of Michigan(密歇根大学统计系)
Frequency-adaptive tensor neural networks for high-dimensional multi-scale problems
SKLMS, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, PR China(数学与系统科学研究院,中国科学院,北京,100190,中国);School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100190, PR China(中国科学院大学数学科学学院,北京,100190,中国);College of Mathematics and Statistics, Chongqing University(重庆大学数学与统计学院)
Multicriteria decision support employing adaptive prediction in a tensor-based feature representation
University of Campinas (UNICAMP)(坎皮纳斯大学);School of Electrical and Computer Engineering (FEEC), University of Campinas (UNICAMP)(坎皮纳斯大学电气与计算机工程学院);School of Applied Sciences (FCA), University of Campinas (UNICAMP)(坎皮纳斯大学应用科学学院)
ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
Shanghai Jiao Tong University(上海交通大学);Shanghai Qi Zhi Institute(上海期智研究院);Microsoft Research(微软研究院);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));University of Rochester(罗切斯特大学)
An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compression
Computational Science Research Center(计算科学研究中心);University of California Irvine(加州大学尔湾分校);University of California, Davis(加州大学戴维斯分校);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Mathematics(数学系);Department of Computer Science(计算机科学系);California State University Fullerton(加州州立大学富尔顿分校)
A Multi-View Coupled Tensor Decomposition for Lightweight Online Adaptive Traffic Prediction
School of Mathematics and Statistics, Central China Normal University(华中师范大学数学与统计学学院);School of Mathematics and Statistics, Beijing Institute of Technology(北京理工大学数学与统计学学院);State Key Laboratory of Mathematical Sciences, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院数学科学国家重点实验室);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院)
Robust Neural Tucker Factorization with Bias Correction and Adaptive Initialization
School of Computer Science and Engineering, Chongqing University of Science and Technology(重庆科技大学计算机科学与工程学院);College of Computer and Information Science, School of Software, Southwest University(西南大学计算机与信息科学学院 软件学院)
Plastic tensor networks for interpretable generative modeling
Institute for Solid State Physics, The University of Tokyo(东京大学固态物理研究所);Graduate School of Informatics, Kyoto University(京都大学信息科学研究生院);Institute for Physics of Intelligence, The University of Tokyo(东京大学智能物理研究所);Trans-scale Quantum Science Institute, The University of Tokyo(东京大学跨尺度量子科学研究所)