We extend the Blahut-Arimoto algorithm for maximizing Massey's directed information. The algorithm can be used for estimating the capacity of channels with delayed feedback, where the feedback is a deterministic function of the output. In order to do so, we apply the ideas from the regular Blahut-Arimoto algorithm, i.e., the alternating maximization procedure, onto our new problem. We provide both upper and lower bound sequences that converge to the optimum value. Our main insight in this paper is that in order to find the maximum of the directed information over causal conditioning probability mass function (PMF), one can use a backward index time maximization combined with the alternating maximization procedure. We give a detailed description of the algorithm, its complexity, the memory needed, and several numerical examples.
In this paper, we investigate the convergence speed of the Arimoto-Blahut algorithm. For many channel matrices the convergence is exponential, but for some channel matrices it is slower than exponential. By analyzing the Taylor expansion of the defining function of the Arimoto-Blahut algorithm, we will make the conditions clear for the exponential or slower convergence. The analysis of the slow convergence is new in this paper. Based on the analysis, we will compare the convergence speed of the Arimoto-Blahut algorithm numerically with the values obtained in our theorems for several channel ma…
The paper first recalls the Blahut Arimoto algorithm for computing the capacity of arbitrary discrete memoryless channels, as an example of an iterative algorithm working with probability density estimates. Then, a geometrical interpretation of this algorithm based on projections onto linear and exponential families of probabilities is provided. Finally, this understanding allows also to propose to write the Blahut-Arimoto algorithm, as a true proximal point algorithm. it is shown that the corresponding version has an improved convergence rate, compared to the initial algorithm, as well as in …
Improved Capacity Upper Bounds for the Deletion Channel using a Parallelized Blahut-Arimoto Algorithm
Instituto Superior Técnico, Universidade de Lisboa（里斯本技术学院，里斯本大学）；Instituto de Telecomunicações and Departamento de Matemática, Instituto Superior Técnico, Universidade de Lisboa（电信研究所与数学系，里斯本技术学院，里斯本大学）
The Arimoto--Blahut algorithm for computing the capacity of a discrete memoryless channel is revisited. A so-called ``squeezing'' strategy is used to design algorithms that preserve its simplicity and monotonic convergence properties, but have provably better rates of convergence.
In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure.With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples.
In this report, we study communication over an additive white Gaussian noise channel with a fixed signal constellation. We measure how much information we can send through this channel and how to improve the rate of communication by changing the input probability distribution. More precisely, we study the mutual information obtained from the Maxwell-Boltzmann distribution, the Blahut-Arimoto algorithm and a constrained version of the Blahut-Arimoto algorithm. We emphasise the fact that the Maxwell--Boltzmann distribution is not optimal. We also observe that the Blahut-Arimoto algorithm does no…
The source coding problem with action-dependent side information at the decoder has recently been introduced to model data acquisition in resource-constrained systems. In this paper, an efficient algorithm for numerical computation of the rate-distortion-cost function for this problem is proposed, and a convergence proof is provided. Moreover, a two-stage code design based on multiplexing is put forth, whereby the first stage encodes the actions and the second stage is composed of an array of classical Wyner-Ziv codes, one for each action. Specific coding/decoding strategies are designed based…
In [8] (Nakagawa, et.al., IEEE Trans. IT, 2021), we investigated the convergence speed of the Arimoto-Blahut algorithm. In [8], the convergence of the order $O(1/N)$ was analyzed by focusing on the second-order nonlinear recurrence formula consisting of the first- and second-order terms of the Taylor expansion of the defining function of the Arimoto-Blahut algorithm. However, in [8], an infinite number of inequalities were assumed as a "conjecture," and proofs were given based on the conjecture. In this paper, we report a proof of the convergence of the order $O(1/N)$ for a class of channel ma…
Based on Arimoto's work in 1978, we propose an iterative algorithm for computing the capacity of a discrete memoryless classical-quantum channel with a finite input alphabet and a finite dimensional output, which we call the Blahut-Arimoto algorithm for classical-quantum channel, and an input cost constraint is considered. We show that to reach $\varepsilon$ accuracy, the iteration complexity of the algorithm is up bounded by $\frac{\log n\log\varepsilon}{\varepsilon}$ where $n$ is the size of the input alphabet. In particular, when the output state $\{ρ_x\}_{x\in \mathcal{X}}$ is linearly ind…
This paper recalls the proximal point method. We study two iterative algorithms: the Blahut-Arimoto algorithm for computing the capacity of arbitrary discrete memoryless channels, as an example of an iterative algorithm working with probability density estimates and the iterative decoding of the Bit Interleaved Coded Modulation (BICM-ID). For these iterative algorithms, we apply the proximal point method which allows new interpretations with improved convergence rate.
The Arimoto algorithm computes the Gallager function $\max_Q {E}_{0}^{}(ρ,Q)$ for a given channel ${P}_{}^{}(y \,|\, x)$ and parameter $ρ$, by means of alternating maximization. Along the way, it generates a sequence of input distributions ${Q}_{1}^{}(x)$, ${Q}_{2}^{}(x)$, ... , that converges to the maximizing input ${Q}_{}^{*}(x)$. We propose a stochastic interpretation for the Arimoto algorithm. We show that for a random (i.i.d.) codebook with a distribution ${Q}_{k}^{}(x)$, the next distribution ${Q}_{k+1}^{}(x)$ in the Arimoto algorithm is equal to the type (${Q}'$) of the feasible transm…
In this paper we introduce two variational equalities of directed information, which are analogous to those of mutual information employed in the Blahut-Arimoto Algorithm (BAA). Subsequently, we introduce nonanticipative Rate Distortion Function (RDF) ${R}^{na}_{0,n}(D)$ defined via directed information introduced in [1], and we establish its equivalence to Gorbunov-Pinsker's nonanticipatory $ε$-entropy $R^{\varepsilon}_{0,n}(D)$. By invoking certain results we first establish existence of the infimizing reproduction distribution for ${R}^{na}_{0,n}(D)$, and then we give its implicit form for …
Directed information or its variants are utilized extensively in the characterization of the capacity of channels with memory and feedback, nonanticipative lossy data compression, and their generalizations to networks. In this paper, we derive several functional and topological properties of directed information for general abstract alphabets (complete separable metric spaces) using the topology of weak convergence of probability measures. These include convexity of the set of consistent distributions, which uniquely define causally conditioned distributions, convexity and concavity of directe…
Rate-distortion (R-D) function, a key quantity in information theory, characterizes the fundamental limit of how much a data source can be compressed subject to a fidelity criterion, by any compression algorithm. As researchers push for ever-improving compression performance, establishing the R-D function of a given data source is not only of scientific interest, but also sheds light on the possible room for improving compression algorithms. Previous work on this problem relied on distributional assumptions on the data source (Gibson, 2017) or only applied to discrete data (Blahut, 1972; Arimo…
A Constrained BA Algorithm for Rate-Distortion and Distortion-Rate Functions
Tsinghua University（清华大学）；Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China（电子科技大学长三角研究院（湖州））；Huawei Technologies Co. Ltd.（华为技术有限公司）；University of Science and Technology of China（中国科学技术大学）
本文改进 Blahut-Arimoto 算法，通过 Newton 求根动态更新乘子，直接计算给定失真下的 RD 及 DR 函数，并证明其 O(1/n) 收敛率与运算复杂度，实验显示显著加速。
