A new message-passing (MP) method is considered for the matrix completion problem associated with recommender systems. We attack the problem using a (generative) factor graph model that is related to a probabilistic low-rank matrix factorization. Based on the model, we propose a new algorithm, termed IMP, for the recovery of a data matrix from incomplete observations. The algorithm is based on a clustering followed by inference via MP (IMP). The algorithm is compared with a number of other matrix completion algorithms on real collaborative filtering (e.g., Netflix) data matrices. Our results show that, while many methods perform similarly with a large number of revealed entries, the IMP algorithm outperforms all others when the fraction of observed entries is small. This is helpful because it reduces the well-known cold-start problem associated with collaborative filtering (CF) systems in practice.
Artificial Intelligence Technology Engineering Research Center, Open University of Henan（河南开放大学人工智能技术工程研究中心）；University of Wollongong（伍伦贡大学）；Weizmann Institute of Science（魏茨曼科学研究所）；University of Sydney（悉尼大学）
We consider matrix factorization (MF) with certain constraints, which finds wide applications in various areas. Leveraging variational inference (VI) and unitary approximate message passing (UAMP), we develop a Bayesian approach to MF with an efficient message passing implementation, called UAMPMF. With proper priors imposed on the factor matrices, UAMPMF can be used to solve many problems that can be formulated as MF, such as non negative matrix factorization, dictionary learning, compressive sensing with matrix uncertainty, robust principal component analysis, and sparse matrix factorization…
Graph convolution networks, like message passing graph convolution networks (MPGCNs), have been a powerful tool in representation learning of networked data. However, when data is heterogeneous, most architectures are limited as they employ a single strategy to handle multi-channel graph signals and they typically focus on low-frequency information. In this paper, we present a novel graph convolution operator, termed BankGCN, which keeps benefits of message passing models, but extends their capabilities beyond `low-pass' features. It decomposes multi-channel signals on graphs into subspaces an…
In recent years, contrastive learning achieves impressive results on self-supervised visual representation learning, but there still lacks a rigorous understanding of its learning dynamics. In this paper, we show that if we cast a contrastive objective equivalently into the feature space, then its learning dynamics admits an interpretable form. Specifically, we show that its gradient descent corresponds to a specific message passing scheme on the corresponding augmentation graph. Based on this perspective, we theoretically characterize how contrastive learning gradually learns discriminative f…
This paper introduces a novel message-passing (MP) framework for the collaborative filtering (CF) problem associated with recommender systems. We model the movie-rating prediction problem popularized by the Netflix Prize, using a probabilistic factor graph model and study the model by deriving generalization error bounds in terms of the training error. Based on the model, we develop a new MP algorithm, termed IMP, for learning the model. To show superiority of the IMP algorithm, we compare it with the closely related expectation-maximization (EM) based algorithm and a number of other matrix co…
An improved inference method for densely connected systems is presented. The approach is based on passing condensed messages between variables, representing macroscopic averages of microscopic messages. We extend previous work that showed promising results in cases where the solution space is contiguous to cases where fragmentation occurs. We apply the method to the signal detection problem of Code Division Multiple Access (CDMA) for demonstrating its potential. A highly efficient practical algorithm is also derived on the basis of insight gained from the analysis.
Variational message passing (VMP), belief propagation (BP) and expectation propagation (EP) have found their wide applications in complex statistical signal processing problems. In addition to viewing them as a class of algorithms operating on graphical models, this paper unifies them under an optimization framework, namely, Bethe free energy minimization with differently and appropriately imposed constraints. This new perspective in terms of constraint manipulation can offer additional insights on the connection between different message passing algorithms and is valid for a generic statistic…
We introduce an iterative solution to the problem of interference alignment (IA) over MIMO channels based on a message-passing formulation. We propose a parameterization of the messages that enables the computation of IA precoders by a min-sum algorithm over continuous variable spaces -- under this parameterization, suitable approximations of the messages can be computed in closed-form. We show that the iterative leakage minimization algorithm of Cadambe et al. is a special case of our message-passing algorithm, obtained for a particular schedule. Finally, we show that the proposed algorithm c…
The ubiquity of approximately sparse data has led a variety of com- munities to great interest in compressed sensing algorithms. Although these are very successful and well understood for linear measurements with additive noise, applying them on real data can be problematic if imperfect sensing devices introduce deviations from this ideal signal ac- quisition process, caused by sensor decalibration or failure. We propose a message passing algorithm called calibration approximate message passing (Cal-AMP) that can treat a variety of such sensor-induced imperfections. In addition to deriving the…
We consider dictionary learning and blind calibration for signals and matrices created from a random ensemble. We study the mean-squared error in the limit of large signal dimension using the replica method and unveil the appearance of phase transitions delimiting impossible, possible-but-hard and possible inference regions. We also introduce an approximate message passing algorithm that asymptotically matches the theoretical performance, and show through numerical tests that it performs very well, for the calibration problem, for tractable system sizes.
The decoding performance of product codes and staircase codes based on iterative bounded-distance decoding (iBDD) can be improved with the aid of a moderate amount of soft information, maintaining a low decoding complexity. One promising approach is error-and-erasure (EaE) decoding, whose performance can be reliably estimated with density evolution (DE). However, the extrinsic message passing (EMP) decoder required by the DE analysis entails a much higher complexity than the simple intrinsic message passing (IMP) decoder. In this paper, we simplify the EMP decoding algorithm for the EaE channe…
The Consultative Committee for Space Data Systems (CCSDS) 141.11-O-1 Line Product Code (LPC) provides a rare opportunity to compare maximum-likelihood decoding and message passing. The LPC considered in this paper is intended to serve as the inner code in conjunction with a (255,239) Reed Solomon (RS) code whose symbols are bytes of data. This paper represents the 141.11-O-1 LPC as a bipartite graph and uses that graph to formulate both maximum likelihood (ML) and message passing algorithms. ML decoding must, of course, have the best frame error rate (FER) performance. However, a fixed point i…
Normal priors with unknown variance (NUV) have long been known to promote sparsity and to blend well with parameter learning by expectation maximization (EM). In this paper, we advocate this approach for linear state space models for applications such as the estimation of impulsive signals, the detection of localized events, smoothing with occasional jumps in the state space, and the detection and removal of outliers. The actual computations boil down to multivariate-Gaussian message passing algorithms that are closely related to Kalman smoothing. We give improved tables of Gaussian-message co…
In this paper, we consider quantized decoding of LDPC codes on the binary symmetric channel. The binary message passing algorithms, while allowing extremely fast hardware implementation, are not very attractive from the perspective of performance. More complex decoders such as the ones based on belief propagation exhibit superior performance but lead to slower decoders. The approach in this paper is to consider message passing decoders that have larger message alphabet (thereby providing performance improvement) as well as low complexity (thereby ensuring fast decoding). We propose a class of …
We consider the problem of recovering a block (or group) sparse signal from an underdetermined set of random linear measurements, which appear in compressed sensing applications such as radar and imaging. Recent results of Donoho, Johnstone, and Montanari have shown that approximate message passing (AMP) in combination with Stein's shrinkage outperforms group LASSO for large block sizes. In this paper, we prove that, for a fixed block size and in the strong undersampling regime (i.e., having very few measurements compared to the ambient dimension), AMP cannot improve upon group LASSO, thereby …
We propose a verification-based Interval-Passing (IP) algorithm for iteratively reconstruction of nonnegative sparse signals using parity check matrices of low-density parity check (LDPC) codes as measurement matrices. The proposed algorithm can be considered as an improved IP algorithm by further incorporation of the mechanism of verification algorithm. It is proved that the proposed algorithm performs always better than either the IP algorithm or the verification algorithm. Simulation results are also given to demonstrate the superior performance of the proposed algorithm.
This letter proposes a novel message-passing algorithm for signal recovery in compressed sensing. The proposed algorithm solves the disadvantages of approximate message-passing (AMP) and orthogonal/vector AMP, and realizes their advantages. AMP converges only in a limited class of sensing matrices while it has low complexity. Orthogonal/vector AMP requires a high-complexity matrix inversion while it is applicable for a wide class of sensing matrices. The key feature of the proposed algorithm is the so-called Onsager correction via a convolution of messages in all preceding iterations while the…
Cloud Radio Access Network (C-RAN) is a promising architecture for unprecedented capacity enhancement in next-generation wireless networks thanks to the centralization and virtualization of base station processing. However, centralized signal processing in C-RANs involves high computational complexity that quickly becomes unaffordable when the network grows to a huge size. Among the first, this paper endeavours to design a scalable uplink signal detection algorithm, in the sense that both the complexity per unit network area and the total computation time remain constant when the network size …
In this paper, we propose a practical receiver for multicarrier signals subjected to a strong memoryless nonlinearity. The receiver design is based on a generalized approximate message passing (GAMP) framework, and this allows real-time algorithm implementation in software or hardware with moderate complexity. We demonstrate that the proposed receiver can provide more than a 2dB gain compared with an ideal uncoded linear OFDM transmission at a BER range $10^{-4}\div10^{-6}$ in the AWGN channel, when the OFDM signal is subjected to clipping nonlinearity and the crest-factor of the clipped wavef…
In this paper we consider Basis Pursuit De-Noising (BPDN) problems in which the sparse original signal is drawn from a finite alphabet. To solve this problem we propose an iterative message passing algorithm, which capitalises not only on the sparsity but by means of a prior distribution also on the discrete nature of the original signal. In our numerical experiments we test this algorithm in combination with a Rademacher measurement matrix and a measurement matrix derived from the random demodulator, which enables compressive sampling of analogue signals. Our results show in both cases signif…
For the problem of multi-class linear classification and feature selection, we propose approximate message passing approaches to sparse multinomial logistic regression (MLR). First, we propose two algorithms based on the Hybrid Generalized Approximate Message Passing (HyGAMP) framework: one finds the maximum a posteriori (MAP) linear classifier and the other finds an approximation of the test-error-rate minimizing linear classifier. Then we design computationally simplified variants of these two algorithms. Next, we detail methods to tune the hyperparameters of their assumed statistical models…
Vector Approximate Message Passing (VAMP) provides the means of solving a linear inverse problem in a Bayes-optimal way assuming the measurement operator is sufficiently random. However, VAMP requires implementing the linear minimum mean squared error (LMMSE) estimator at every iteration, which makes the algorithm intractable for large-scale problems. In this work, we present a class of warm-started (WS) methods that provides a scalable approximation of LMMSE within VAMP. We show that a Message Passing (MP) algorithm equipped with a method from this class can converge to the fixed point of VAM…
We consider an $\ell_2$-regularized non-convex optimization problem for recovering signals from their noisy phaseless observations. We design and study the performance of a message passing algorithm that aims to solve this optimization problem. We consider the asymptotic setting $m,n \rightarrow \infty$, $m/n \rightarrow δ$ and obtain sharp performance bounds, where $m$ is the number of measurements and $n$ is the signal dimension. We show that for complex signals the algorithm can perform accurate recovery with only $m=\left ( \frac{64}{π^2}-4\right)n\approx 2.5n$ measurements. Also, we provi…
We present a novel compressed sensing recovery algorithm - termed Bayesian Optimal Structured Signal Approximate Message Passing (BOSSAMP) - that jointly exploits the prior distribution and the structured sparsity of a signal that shall be recovered from noisy linear measurements. Structured sparsity is inherent to group sparse and jointly sparse signals. Our algorithm is based on approximate message passing that poses a low complexity recovery algorithm whose Bayesian optimal version allows to specify a prior distribution for each signal component. We utilize this feature in order to establis…
The design of message passing (MP) algorithms on factor graphs is an effective manner to implement channel estimation (CE) in wireless communication systems, which performance can be further improved by exploiting prior probability models that accurately match the channel characteristics. In this work, we study the CE problem in a downlink massive multiple-input multiple-output (MIMO) orthogonal frequency division multiplexing (OFDM) system. As the prior probability, we propose the Markov chain two-state Gaussian mixture with large variance differences (TSGM-LVD) model to exploit the structure…
Sparse Bayesian learning (SBL) can be implemented with low complexity based on the approximate message passing (AMP) algorithm. However, it is vulnerable to `difficult' measurement matrices as AMP can easily diverge. Damped AMP has been used to alleviate the problem at the cost of slowing the convergence speed. In this work, we propose an SBL algorithm based on the AMP with unitary transformation (UTAMP), where the shape parameter of the hyperprior is tuned automatically. It is shown that, compared to the state-of-the-art AMP based SBL algorithm, the proposed UTAMP-SBL is much more robust and …
Sparse regression codes with approximate message passing (AMP) decoding have gained much attention in recent times. The concepts underlying this coding scheme extend to unsourced access with coded compressed sensing (CCS), as first pointed out by Fengler, Jung, and Caire. More specifically, their approach uses a concatenated coding framework with an inner AMP decoder followed by an outer tree decoder. In the original implementation, these two components work independently of each other, with the tree decoder acting on the static output of the AMP decoder. This article introduces a novel framew…
In this paper, we address the message-passing receiver design for the 3D massive MIMO-OFDM systems. With the aid of the central limit argument and Taylor-series approximation, a computationally efficient receiver that performs joint channel estimation and decoding is devised by the framework of expectation propagation. Specially, the local belief defined at the channel transition function is expanded up to the second order with Wirtinger calculus, to transform the messages sent by the channel transition function to a tractable form. As a result, the channel impulse response (CIR) between each …
The topological interference management (TIM) problem studies partially-connected interference networks with no channel state information except for the network topology (i.e., connectivity graph) at the transmitters. In this paper, we consider a similar problem in the uplink cellular networks, while message passing is enabled at the receivers (e.g., base stations), so that the decoded messages can be routed to other receivers via backhaul links to help further improve network performance. For this TIM problem with decoded message passing (TIM-MP), we model the interference pattern by conflict…
Approximate message passing (AMP) methods have gained recent traction in sparse signal recovery. Additional information about the signal, or \emph{side information} (SI), is commonly available and can aid in efficient signal recovery. This work presents an AMP-based framework that exploits SI and can be readily implemented in various settings for which the SI results in separable distributions. To illustrate the simplicity and applicability of our approach, this framework is applied to a Bernoulli-Gaussian (BG) model and a time-varying birth-death-drift (BDD) signal model, motivated by applica…
Solving a large-scale regularized linear inverse problem using multiple processors is important in various real-world applications due to the limitations of individual processors and constraints on data sharing policies. This paper focuses on the setting where the matrix is partitioned column-wise. We extend the algorithmic framework and the theoretical analysis of approximate message passing (AMP), an iterative algorithm for solving linear inverse problems, whose asymptotic dynamics are characterized by state evolution (SE). In particular, we show that column-wise multiprocessor AMP (C-MP-AMP…
Due to the massive number of devices in the M2M communication era, new challenges have been brought to the existing random-access (RA) mechanism, such as severe preamble collisions and resource block (RB) wastes. To address these problems, a novel sparse message passing (SMP) algorithm is proposed, based on a factor graph on which Bernoulli messages are updated. The SMP enables an accurate estimation on the activity of the devices and the identity of the preamble chosen by each active device. Aided by the estimation, the RB efficiency for the uplink data transmission can be improved, especiall…
We address noisy message-passing decoding of lowdensity parity-check (LDPC) codes over additive white Gaussian noise channels. Message-passing decoders in which certain processing units iteratively exchange messages are common for decoding LDPC codes. The exchanged messages are in general subject to internal noise in hardware implementation of these decoders. We model the internal decoder noise as additive white Gaussian noise (AWGN) degrading exchanged messages. Using Gaussian approximation of the exchanged messages, we perform a two-dimensional density evolution analysis for the noisy LDPC d…
Cognitive Radio has been proposed as a key technology to significantly improve spectrum usage in wireless networks by enabling unlicensed users to access unused resource. We present new algorithms that are needed for the implementation of opportunistic scheduling policies that maximize the throughput utilization of resources by secondary users, under maximum interference constraints imposed by existing primary users. Our approach is based on the Belief Propagation (BP) algorithm, which is advantageous due to its simplicity and potential for distributed implementation. We examine convergence pr…
The orthogonal-time-frequency-space (OTFS) modulation has emerged as a promising modulation scheme for high mobility wireless communications. To harvest the time and frequency diversity promised by OTFS, some promising detectors, especially message passing based ones, have been developed by taking advantage of the sparsity of the channel in the delay-Doppler domain. However, when the number of channel paths is relatively large or fractional Doppler {shifts have} to be considered, the complexity of existing detectors is a concern, and the message passing based detectors may suffer from performa…
Bayesian approximate message passing (BAMP) is an efficient method in compressed sensing that is nearly optimal in the minimum mean squared error (MMSE) sense. Bayesian approximate message passing (BAMP) performs joint recovery of multiple vectors with identical support and accounts for correlations in the signal of interest and in the noise. In this paper, we show how to reduce the complexity of vector BAMP via a simple joint decorrelation diagonalization) transform of the signal and noise vectors, which also facilitates the subsequent performance analysis. We prove that BAMP and the correspo…
Faced with the massive connection, sporadic transmission, and small-sized data packets in future cellular communication, a grant-free non-orthogonal random access (NORA) system is considered in this paper, which could reduce the access delay and support more devices. In order to address the joint user activity detection (UAD) and channel estimation (CE) problem in the grant-free NORA system, we propose a deep neural network-aided message passing-based block sparse Bayesian learning (DNN-MP-BSBL) algorithm. In this algorithm, the message passing process is transferred from a factor graph to a d…
Being an effective non-orthogonal multiple access (NOMA) technique, sparse code multiple access (SCMA) is promising for future wireless communication. Compared with orthogonal techniques, SCMA enjoys higher overloading tolerance and lower complexity because of its sparsity. In this paper, based on deterministic message passing algorithm (DMPA), algorithmic simplifications such as domain changing and probability approximation are applied for SCMA decoding. Early termination, adaptive decoding, and initial noise reduction are also employed for faster convergence and better performance. Numerical…
For certain sensing matrices, the Approximate Message Passing (AMP) algorithm efficiently reconstructs undersampled signals. However, in Magnetic Resonance Imaging (MRI), where Fourier coefficients of a natural image are sampled with variable density, AMP encounters convergence problems. In response we present an algorithm based on Orthogonal AMP constructed specifically for variable density partial Fourier sensing matrices. For the first time in this setting a state evolution has been observed. A practical advantage of state evolution is that Stein's Unbiased Risk Estimate (SURE) can be effec…
