Tensor Networks Meet Neural Networks: A Survey and Future Perspectives
City University of Hong Kong（香港城市大学）；Harbin Institute of Technology Shenzhen（哈尔滨工业大学（深圳））；Fudan University（复旦大学）；Quantum Science Center of Guangdong-Hong Kong-Macao Greater Bay Area（粤港澳大湾区量子科学中心）；Information Engineering University（信息工程大学）；Imperial College London（伦敦帝国学院）；Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）；Artificial Intelligence Project, Riken（理化学研究所人工智能项目）
Tensor networks (TNs) and neural networks (NNs) are two fundamental data modeling approaches. TNs were introduced to solve the curse of dimensionality in large-scale tensors by converting an exponential number of dimensions to polynomial complexity. As a result, they have attracted significant attention in the fields of quantum physics and machine learning. Meanwhile, NNs have displayed exceptional performance in various applications, e.g., computer vision, natural language processing, and robotics research. Interestingly, although these two types of networks originate from different observations, they are inherently linked through the typical multilinearity structure underlying both TNs and NNs, thereby motivating a significant number of developments regarding combinations of TNs and NNs. In this paper, we refer to these combinations as tensorial neural networks~(TNNs) and present an introduction to TNNs from both data processing and model architecture perspectives. From the data perspective, we explore the capabilities of TNNs in multi-source fusion, multimodal pooling, data compression, multi-task training, and quantum data processing. From the model perspective, we examine TNNs' integration with various architectures, including Convolutional Neural Networks, Recurrent Neural Networks, Graph Neural Networks, Transformers, Large Language Models, and Quantum Neural Networks. Furthermore, this survey also explores methods for improving TNNs, examines flexible toolboxes for implementing TNNs, and documents TNN development while highlighting potential future directions. To the best of our knowledge, this is the first comprehensive survey that bridges the connections among NNs and TNs. We provide a curated list of TNNs at https://github.com/tnbar/awesome-tensorial-neural-networks.
Large Language Models Meet Graph Neural Networks: A Perspective of Graph Mining
School of Computer Science and Engineering（计算机科学与工程学院）；University of Electronic Science and Technology of China（电子科技大学）；th Research Institute of CETC（中国电子科技集团公司第五十四研究所）
UFRGS（里约热内卢联邦大学）；City, University of London（伦敦城市大学）；University of Siena（锡耶纳大学）；Université Côte d’Azur（蔚蓝海岸大学）；Rice University（莱斯大学）
Neural-symbolic computing has now become the subject of interest of both academic and industry research laboratories. Graph Neural Networks (GNN) have been widely used in relational and symbolic domains, with widespread application of GNNs in combinatorial optimization, constraint satisfaction, relational reasoning and other scientific domains. The need for improved explainability, interpretability and trust of AI systems in general demands principled methodologies, as suggested by neural-symbolic computing. In this paper, we review the state-of-the-art on the use of GNNs as a model of neural-…
Caltech（加州理工学院）；University of Southern California（南加州大学）
Low-rank tensor regression, a new model class that learns high-order correlation from data, has recently received considerable attention. At the same time, Gaussian processes (GP) are well-studied machine learning models for structure learning. In this paper, we demonstrate interesting connections between the two, especially for multi-way data analysis. We show that low-rank tensor regression is essentially learning a multi-linear kernel in Gaussian processes, and the low-rank assumption translates to the constrained Bayesian inference problem. We prove the oracle inequality and derive the ave…
Sorbonne Universitas（索邦大学）；University of Pittsburgh/Medical Center（匹兹堡大学医学中心）；Carnegie Mellon University（卡内基梅隆大学）
We introduce in this paper the principle of Deep Temporal Networks that allow to add time to convolutional networks by allowing deep integration principles not only using spatial information but also increasingly large temporal window. The concept can be used for conventional image inputs but also event based data. Although inspired by the architecture of brain that inegrates information over increasingly larger spatial but also temporal scales it can operate on conventional hardware using existing architectures. We introduce preliminary results to show the efficiency of the method. More in-de…
Machine learning (ML) is becoming increasingly crucial in many fields of engineering but has not yet played out its full potential in bioprocess engineering. While experimentation has been accelerated by increasing levels of lab automation, experimental planning and data modeling are still largerly depend on human intervention. ML can be seen as a set of tools that contribute to the automation of the whole experimental cycle, including model building and practical planning, thus allowing human experts to focus on the more demanding and overarching cognitive tasks. First, probabilistic programm…
With the ever-improving computing capabilities and storage capacities of mobile devices in line with evolving telecommunication network paradigms, there has been an explosion of research interest towards exploring Distributed Learning (DL) frameworks to realize stringent key performance indicators (KPIs) that are expected in next-generation/6G cellular networks. In conjunction with Edge Computing, Federated Learning (FL) has emerged as the DL architecture of choice in prominent wireless applications. This article lays an outline of how DL in general and FL-based strategies specifically can con…
We study tensor completion in the agnostic setting. In the classical tensor completion problem, we receive $n$ entries of an unknown rank-$r$ tensor and wish to exactly complete the remaining entries. In agnostic tensor completion, we make no assumption on the rank of the unknown tensor, but attempt to predict unknown entries as well as the best rank-$r$ tensor. For agnostic learning of third-order tensors with the square loss, we give the first polynomial time algorithm that obtains a "fast" (i.e., $O(1/n)$-type) rate improving over the rate obtained by reduction to matrix completion. Our pre…
When Bayesian Tensor Completion Meets Multioutput Gaussian Processes: Functional Universality and Rank Learning
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Zhejiang Provincial Key Laboratory of Multi-Modal Communication Networks and Intelligent Information Processing（浙江省多模态通信网络与智能信息处理重点实验室）；School of Science & Engineering, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）科学与工程学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Technical University of Denmark（丹麦技术大学）；NoiseLab, University of California, San Diego（加州大学圣地亚哥分校NoiseLab）；HERON - Center of Excellence in Robotics, Athena R.C（HERON-机器人卓越中心）
Southeast University（东南大学）；City University of Hong Kong（香港城市大学）
This paper explores the problem of clustering ensemble, which aims to combine multiple base clusterings to produce better performance than that of the individual one. The existing clustering ensemble methods generally construct a co-association matrix, which indicates the pairwise similarity between samples, as the weighted linear combination of the connective matrices from different base clusterings, and the resulting co-association matrix is then adopted as the input of an off-the-shelf clustering algorithm, e.g., spectral clustering. However, the co-association matrix may be dominated by po…
Boston College（波士顿学院）；Brandeis University（布兰迪斯大学）；Boston University（波士顿大学）；J.P. Morgan（摩根大通）
We analyze the problem of high-order polynomial approximation from a many-body physics perspective, and demonstrate the descriptive power of entanglement entropy in capturing model capacity and task complexity. Instantiated with a high-order nonlinear dynamics modeling problem, tensor-network models are investigated and exhibit promising modeling advantages. This novel perspective establish a connection between quantum information and functional approximation, which worth further exploration in future research.
Transfer entropy is capable of capturing nonlinear source-destination relations between multi-variate time series. It is a measure of association between source data that are transformed into destination data via a set of linear transformations between their probability mass functions. The resulting tensor formalism is used to show that in specific cases, e.g., in the case the system consists of three stochastic processes, bivariate analysis suffices to distinguish true relations from false relations. This allows us to determine the causal structure as far as encoded in the probability mass fu…
Multi-domain learning aims to benefit from simultaneously learning across several different but related domains. In this chapter, we propose a single framework that unifies multi-domain learning (MDL) and the related but better studied area of multi-task learning (MTL). By exploiting the concept of a \emph{semantic descriptor} we show how our framework encompasses various classic and recent MDL/MTL algorithms as special cases with different semantic descriptor encodings. As a second contribution, we present a higher order generalisation of this framework, capable of simultaneous multi-task-mul…
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代国家重点实验室（注：此处按原文直译，实际常见译名为“南京大学计算机软件新技术国家重点实验室”））；University of Wollongong（卧龙岗大学）
SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective
School of Information Science and Technology, Southwest Jiaotong University（西南交通大学信息科学与技术学院）；School of Mathematical Sciences, University of Electronic Science and Technology of China（电子科技大学数学科学学院）；School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；RIKEN Center for Advanced Intelligence Project (AIP)（理化学研究所先进智能研究中心）
Geometric Algebra Meets Cartesian Tensors: Higher-Order Equivariance for Interatomic Potentials
Dept. of Electrical & Computer Eng.（电气与计算机工程系）；Texas A&M Univ.（德克萨斯A&M大学）；Dept. of Prosthetics & Orthotics（假肢与矫形器系）；College of Science & Eng.（科学与工程学院）；Ankara Univ.（安卡拉大学）；Hamad Bin Khalifa Univ. and Texas A&M Univ. at Qatar（哈马德·本·卡西姆大学和德克萨斯A&M大学（多哈））
Tensor completion is a problem of filling the missing or unobserved entries of partially observed tensors. Due to the multidimensional character of tensors in describing complex datasets, tensor completion algorithms and their applications have received wide attention and achievement in areas like data mining, computer vision, signal processing, and neuroscience. In this survey, we provide a modern overview of recent advances in tensor completion algorithms from the perspective of big data analytics characterized by diverse variety, large volume, and high velocity. We characterize these advanc…
IRIT（图卢兹计算机科学研究所）；Université de Toulouse（图卢兹大学）；Toulouse INP（图卢兹国立理工学院）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；GIPSA-lab（格勒诺布尔图像、信号、自动化与控制实验室）
Tensor models play an increasingly prominent role in many fields, notably in machine learning. In several applications, such as community detection, topic modeling and Gaussian mixture learning, one must estimate a low-rank signal from a noisy tensor. Hence, understanding the fundamental limits of estimators of that signal inevitably calls for the study of random tensors. Substantial progress has been recently achieved on this subject in the large-dimensional limit. Yet, some of the most significant among these results--in particular, a precise characterization of the abrupt phase transition (…
Guangdong University of Technology（广东工业大学）；RIKEN（理化学研究所）
Efficient and accurate low-rank approximation (LRA) methods are of great significance for large-scale data analysis. Randomized tensor decompositions have emerged as powerful tools to meet this need, but most existing methods perform poorly in the presence of noise interference. Inspired by the remarkable performance of randomized block Krylov iteration (rBKI) in reducing the effect of tail singular values, this work designs an rBKI-based Tucker decomposition (rBKI-TK) for accurate approximation, together with a hierarchical tensor ring decomposition based on rBKI-TK for efficient compression …
The Swiss AI Lab - IDSIA（瑞士人工智能实验室（IDSIA））；USI（瑞士意大利语区大学）；SUPSI（瑞士南部应用科学与艺术大学）；NNAISENSE（NNAISENSE公司）
In order to meet the diverse challenges in solving many real-world problems, an intelligent agent has to be able to dynamically construct a model of its environment. Objects facilitate the modular reuse of prior knowledge and the combinatorial construction of such models. In this work, we argue that dynamically bound features (objects) do not simply emerge in connectionist models of the world. We identify several requirements that need to be fulfilled in overcoming this limitation and highlight corresponding inductive biases.
