We consider tensors in the Hierarchical Tucker format and suppose the tensor data to be distributed among several compute nodes. We assume the compute nodes to be in a one-to-one correspondence with the nodes of the Hierarchical Tucker format such that connected nodes can communicate with each other. An appropriate tree structure in the Hierarchical Tucker format then allows for the parallelization of basic arithmetic operations between tensors with a parallel runtime which grows like $\log(d)$, where $d$ is the tensor dimension. We introduce parallel algorithms for several tensor operations, some of which can be applied to solve linear equations $\mathcal{A}X=B$ directly in the Hierarchical Tucker format using iterative methods like conjugate gradients or multigrid. We present weak scaling studies, which provide evidence that the runtime of our algorithms indeed grows like $\log(d)$. Furthermore, we present numerical experiments in which we apply our algorithms to solve a parameter-dependent diffusion equation in the Hierarchical Tucker format by means of a multigrid algorithm.
Big data analysis has become a crucial part of new emerging technologies such as the internet of things, cyber-physical analysis, deep learning, anomaly detection, etc. Among many other techniques, dimensionality reduction plays a key role in such analyses and facilitates feature selection and feature extraction. Randomized algorithms are efficient tools for handling big data tensors. They accelerate decomposing large-scale data tensors by reducing the computational complexity of deterministic algorithms and the communication among different levels of the memory hierarchy, which is the main bo…
Approximating higher-order tensors by the Tucker format has been applied in many fields such as psychometrics, chemometrics, signal processing, pattern classification, and so on. In this paper, we propose some new Tucker-like approximations based on the modal semi-tensor product (STP), especially, a new singular value decomposition (SVD) and a new higher-order SVD (HOSVD) are derived. Algorithms for computing new decompositions are provided. We also give some numerical examples to illustrate our theoretical results.
Low-rank approximation of tensors has been widely used in high-dimensional data analysis. It usually involves singular value decomposition (SVD) of large-scale matrices with high computational complexity. Sketching is an effective data compression and dimensionality reduction technique applied to the low-rank approximation of large matrices. This paper presents two practical randomized algorithms for low-rank Tucker approximation of large tensors based on sketching and power scheme, with a rigorous error-bound analysis. Numerical experiments on synthetic and real-world tensor data demonstrate …
We derive a CUR-type factorization for tensors in the Tucker format based on interpolatory decomposition, which we will denote as Higher Order Interpolatory Decomposition (HOID). Given a tensor $\mathcal{X}$, the algorithm provides a set of column vectors $\{ \mathbf{C}_n\}_{n=1}^d$ which are columns extracted from the mode-$n$ tensor unfolding, along with a core tensor $\mathcal{G}$ and together, they satisfy some error bounds. Compared to the Higher Order SVD (HOSVD) algorithm, the HOID provides a decomposition that preserves certain important features of the original tensor such as sparsity…
This paper presents an extension of Correspondence Analysis (CA) to tensors through High Order Singular Value Decomposition (HOSVD) from a geometric viewpoint. Correspondence analysis is a well-known tool, developed from principal component analysis, for studying contingency tables. Different algebraic extensions of CA to multi-way tables have been proposed over the years, nevertheless neglecting its geometric meaning. Relying on the Tucker model and the HOSVD, we propose a direct way to associate with each tensor mode a point cloud. We prove that the point clouds are related to each other. Sp…
In recent years, the application of tensors has become more widespread in fields that involve data analytics and numerical computation. Due to the explosive growth of data, low-rank tensor decompositions have become a powerful tool to harness the notorious curse of dimensionality. The main forms of tensor decomposition include CP decomposition, Tucker decomposition, tensor train (TT) decomposition, etc. Each of the existing TT decomposition algorithms, including the TT-SVD and randomized TT-SVD, is successful in the field, but neither can both accurately and efficiently decompose large-scale s…
We propose a new algorithm for the computation of a singular value decomposition (SVD) low-rank approximation of a matrix in the Matrix Product Operator (MPO) format, also called the Tensor Train Matrix format. Our tensor network randomized SVD (TNrSVD) algorithm is an MPO implementation of the randomized SVD algorithm that is able to compute dominant singular values and their corresponding singular vectors. In contrast to the state-of-the-art tensor-based alternating least squares SVD (ALS-SVD) and modified alternating least squares SVD (MALS-SVD) matrix approximation methods, TNrSVD can be u…
Many applications in data science and scientific computing involve large-scale datasets that are expensive to store and compute with, but can be efficiently compressed and stored in an appropriate tensor format. In recent years, randomized matrix methods have been used to efficiently and accurately compute low-rank matrix decompositions. Motivated by this success, we focus on developing randomized algorithms for tensor decompositions in the Tucker representation. Specifically, we present randomized versions of two well-known compression algorithms, namely, HOSVD and STHOSVD. We present a detai…
This work is concerned with approximating a trivariate function defined on a tensor-product domain via function evaluations. Combining tensorized Chebyshev interpolation with a Tucker decomposition of low multilinear rank yields function approximations that can be computed and stored very efficiently. The existing Chebfun3 algorithm [Hashemi and Trefethen, SIAM J. Sci. Comput., 39 (2017)]uses a similar format but the construction of the approximation proceeds indirectly, via a so called slice-Tucker decomposition. As a consequence, Chebfun3 sometimes uses unnecessarily many function evaluation…
Efficient Sketching-Based Summation of Tucker Tensors
Department of Mathematics, Virginia Tech（弗吉尼亚理工大学数学系）；Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校奥登计算工程与科学研究所）；Computational Mathematics Group, Rutherford Appleton Laboratory（拉瑟福德·阿普尔顿实验室计算数学组）
In the paper `On the Dirac-Frenkel Variational Principle on Tensor Banach Spaces', we provided a geometrical description of manifolds of tensors in Tucker format with fixed multilinear (or Tucker) rank in tensor Banach spaces, that allowed to extend the Dirac-Frenkel variational principle in the framework of topological tensor spaces. The purpose of this note is to extend these results to more general tensor formats. More precisely, we provide a new geometrical description of manifolds of tensors in tree-based (or hierarchical) format, also known as tree tensor networks, which are intersection…
In this paper, we propose a conservative low rank tensor method to approximate nonlinear Vlasov solutions. The low rank approach is based on our earlier work (arxiv: 2106.08834). It takes advantage of the fact that the differential operators in the Vlasov equation are tensor friendly, based on which we propose to dynamically and adaptively build up low rank solution basis by adding new basis functions from discretization of the differential equation, and removing basis from a singular value decomposition (SVD)-type truncation procedure. For the discretization, we adopt a high order finite diff…
Tensor decomposition methodologies are proposed to reduce the memory requirement of translation operator tensors arising in the fast multipole method-fast Fourier transform (FMM-FFT)-accelerated surface integral equation (SIE) simulators. These methodologies leverage Tucker, hierarchical Tucker (H-Tucker), and tensor train (TT) decompositions to compress the FFT'ed translation operator tensors stored in three-dimensional (3D) and four-dimensional (4D) array formats. Extensive numerical tests are performed to demonstrate the memory saving achieved by and computational overhead introduced by the…
The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a $d$-dimensi…
Fast multidimensional convolution in low-rank formats via cross approximation
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Institute of Numerical Mathematics, Russian Academy of Sciences（俄罗斯科学院数值数学研究所）
We propose new algorithms for singular value decomposition (SVD) of very large-scale matrices based on a low-rank tensor approximation technique called the tensor train (TT) format. The proposed algorithms can compute several dominant singular values and corresponding singular vectors for large-scale structured matrices given in a TT format. The computational complexity of the proposed methods scales logarithmically with the matrix size under the assumption that both the matrix and the singular vectors admit low-rank TT decompositions. The proposed methods, which are called the alternating lea…
We propose a constructive algorithm that decomposes an arbitrary real tensor into a finite sum of orthonormal rank-1 outer products. The algorithm, named TTr1SVD, works by converting the tensor into a tensor-train rank-1 (TTr1) series via the singular value decomposition (SVD). TTr1SVD naturally generalizes the SVD to the tensor regime with properties such as uniqueness for a fixed order of indices, orthogonal rank-1 outer product terms, and easy truncation error quantification. Using an outer product column table it also allows, for the first time, a complete characterization of all tensors o…
Tucker Tensor analysis of Matern functions in spatial statistics
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Max-Planck Institute for Mathematics in the Sciences（马克斯·普朗克科学促进学会数学研究所）；Max-Planck Institute for Dynamics of Complex Technical Systems（马克斯·普朗克复杂技术系统动力学研究所）；Technische Universitaet Braunschweig（布伦瑞克工业大学）
Geometry-aware training of factorized layers in tensor Tucker format
School of Mathematics, Gran Sasso Science Institute（格兰萨索科学研究所数学系）；Computer Science and Mathematics Division, Oak Ridge National Laboratory（橡树岭国家实验室计算机科学与数学部）；Department of Mathematics, University of Innsbruck（因斯布鲁克大学数学系）；Department of Data Science, Norwegian University of Life Sciences（挪威生命科学大学数据科学系）；School of Mathematics and Maxwell Institute, University of Edinburgh（爱丁堡大学数学系和麦克斯韦研究所）
There is an emerging interest in tensor factorization applications in big-data analytics and machine learning. To speed up the factorization of extra-large datasets, organized in multidimensional arrays (aka tensors), easy to compute compression-based tensor representations, such as Tucker and Tensor Train formats, are used to approximate the initial large-tensor. Further, tensor factorization is used to extract latent features that can facilitate discoveries of new mechanisms and signatures hidden in the data, where the explainability of the latent features is of principal importance. Nonnega…
Low-Rank Tensor Networks for Dimensionality Reduction and Large-Scale Optimization Problems: Perspectives and Challenges PART 1
RIKEN Brain Science Institute（理化学研究所脑科学研究所）；SKOLTECH（斯科尔科沃科学技术学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Institute of Numerical Mathematics of Russian Academy of Sciences（俄罗斯科学院计算数学研究所）；Imperial College（帝国理工学院）
In this era of big data, data analytics and machine learning, it is imperative to find ways to compress large data sets such that intrinsic features necessary for subsequent analysis are not lost. The traditional workhorse for data dimensionality reduction and feature extraction has been the matrix SVD, which presupposes that the data has been arranged in matrix format. Our main goal in this study is to show that high-dimensional data sets are more compressible when treated as tensors (aka multiway arrays) and compressed via tensor-SVDs under the tensor-tensor product structures in (Kilmer and…
Tensor numerical methods, based on the rank-structured tensor representation of $d$-variate functions and operators, are designed to provide $O(dn)$ complexity of numerical calculations on $n^{\otimes d }$ grids contrary to $O(n^d)$ scaling by conventional grid-based methods. However, multiple tensor operations may lead to enormous increase in the tensor ranks (curse of ranks) of the target data, making calculation intractable. Therefore one of the most important steps in tensor calculations is the robust and efficient rank reduction procedure which should be performed many times in the course…
Effective algorithms for tensor train decomposition via the UTV framework
School of Mathematical Sciences, Chengdu University of Technology（成都理工大学数学科学学院）；School of Mathematics and Statistics, Guizhou University（贵州大学数学与统计学院）；School of Mathematical Sciences and Key Laboratory of Mathematics for Nonlinear Sciences, Fudan University（复旦大学数学科学系与非线性科学数学重点实验室）
A New Tensor Network: Tubal Tensor Train and Its Applications
Lab of Machine Learning and Knowledge Representation, Innopolis University（机器学习与知识表示实验室，因诺波利斯大学）；Innopolis University（因诺波利斯大学）；Skolkovo Institute of Science and Technology, Center for Artificial Intelligence Technology（斯克尔克沃科学与技术学院，人工智能技术中心）；Systems Research Institute of Polish Academy of Science（波兰科学院系统研究 institute）
Peking University（北京大学）；Chinese Academy of Sciences（中国科学院）
The low multilinear rank approximation, also known as the truncated Tucker decomposition, has been extensively utilized in many applications that involve higher-order tensors. Popular methods for low multilinear rank approximation usually rely directly on matrix SVD, therefore often suffer from the notorious intermediate data explosion issue and are not easy to parallelize, especially when the input tensor is large. In this paper, we propose a new class of truncated HOSVD algorithms based on alternating least squares (ALS) for efficiently computing the low multilinear rank approximation of ten…
