This paper compares and evaluates a set of non-parametric mutual information estimators with the goal of providing a novel toolset to progress in the analysis of the capacity of the nonlinear optical channel, which is currently an open problem. In the first part of the paper, the methods of the study are presented. The second part details their application to several optically-related channels to highlight their features.
We consider a model nondispersive nonlinear optical fiber channel with additive white Gaussian noise at large $\mathrm{SNR}$ (signal-to-noise ratio) in the intermediate power region. Using Feynman path-integral technique we for the first time find the optimal input signal distribution maximizing the channel's per-sample mutual information. The finding of the optimal input signal distribution allows us to improve previously known estimates for the channel capacity. The output signal entropy, conditional entropy, and per-sample mutual information are calculated for Gaussian, half-Gaussian and mo…
The mutual information between a complex-valued channel input and its complex-valued output is decomposed into four parts based on polar coordinates: an amplitude term, a phase term, and two mixed terms. Numerical results for the additive white Gaussian noise (AWGN) channel with various inputs show that, at high signal-to-noise ratio (SNR), the amplitude and phase terms dominate the mixed terms. For the AWGN channel with a Gaussian input, analytical expressions are derived for high SNR. The decomposition method is applied to partially coherent channels and a property of such channels called "s…
HIIT / Department of Computer Science University of Helsinki（赫尔辛基大学计算机科学系 / 赫尔辛基信息技术研究所）
We propose a method for learning Markov network structures for continuous data without invoking any assumptions about the distribution of the variables. The method makes use of previous work on a non-parametric estimator for mutual information which is used to create a non-parametric test for multivariate conditional independence. This independence test is then combined with an efficient constraint-based algorithm for learning the graph structure. The performance of the method is evaluated on several synthetic data sets and it is shown to learn considerably more accurate structures than compet…
Advanced channel decoders rely on soft-decision decoder inputs for which mutual information (MI) is the natural figure of merit. In this paper, we analyze an optical fiber system by evaluating MI as the maximum achievable rate of transmission of such a system. MI is estimated by means of histograms for which the correct bin number is determined in a blind way. The MI estimate obtained this way shows excellent accuracy in comparison with the true MI of 16-state quadrature amplitude modulation (QAM) over an additive white Gaussian noise channel with additional phase noise, which is a simplified …
In this paper, achievable information rates (AIR) for fiber optical communications are discussed. It is shown that AIRs such as the mutual information and generalized mutual information are good design metrics for coded optical systems. The theoretical predictions of AIRs are compared to the performance of modern codes including low-parity density check (LDPC) and polar codes. Two different computation methods for these AIRs are also discussed: Monte-Carlo integration and Gauss-Hermite quadrature. Closed-form ready-to-use approximations for such computations are provided for arbitrary constell…
Besides the different approaches suggested in the literature, accurate estimation of the order of a Markov chain from a given symbol sequence is an open issue, especially when the order is moderately large. Here, parametric significance tests of conditional mutual information (CMI) of increasing order $m$, $I_c(m)$, on a symbol sequence are conducted for increasing orders $m$ in order to estimate the true order $L$ of the underlying Markov chain. CMI of order $m$ is the mutual information of two variables in the Markov chain being $m$ time steps apart, conditioning on the intermediate variable…
We consider the information channel described by Schrödinger equation with additive Gaussian noise. We introduce the model of the input signal and the model of the output signal receiver. For this channel, using perturbation theory for the small nonlinearity parameter, we calculate the first three terms of the expansion of the conditional probability density function in the nonlinearity parameter. At large signal-to-noise power ratio we calculate the conditional entropy, the output signal entropy, and the mutual information in the leading and next-to-leading order in the nonlinearity parameter…
The Mutual Information (MI) is an often used measure of dependency between two random variables utilized in information theory, statistics and machine learning. Recently several MI estimators have been proposed that can achieve parametric MSE convergence rate. However, most of the previously proposed estimators have the high computational complexity of at least $O(N^2)$. We propose a unified method for empirical non-parametric estimation of general MI function between random vectors in $\mathbb{R}^d$ based on $N$ i.i.d. samples. The reduced complexity MI estimator, called the ensemble dependen…
Fritz Haber Institute of the Max Planck Society（马克斯·普朗克学会弗里茨·哈伯研究所）；Humboldt-Universität zu Berlin（柏林洪堡大学）
The identification of relevant features, i.e., the driving variables that determine a process or the properties of a system, is an essential part of the analysis of data sets with a large number of variables. A mathematical rigorous approach to quantifying the relevance of these features is mutual information. Mutual information determines the relevance of features in terms of their joint mutual dependence to the property of interest. However, mutual information requires as input probability distributions, which cannot be reliably estimated from continuous distributions such as physical quanti…
Using the path-integral technique we examine the mutual information for the communication channel modelled by the nonlinear Schrödinger equation with additive Gaussian noise. The nonlinear Schrödinger equation is one of the fundamental models in nonlinear physics, and it has a broad range of applications, including fiber optical communications --- the backbone of the Internet. At large signal-to-noise ratio ($\mathrm{SNR}$) we present the mutual information through the path-integral which is convenient for the perturbative expansion in nonlinearity. In the limit of small noise and small nonlin…
The maximum achievable rate or mutual informa- tion of multidimensional rotationally invariant distributions in the presence of additive white Gaussian noise is analyzed. A simple expression for the special case of multisphere distributions is derived. Such distributions consist of points in a multidimensional Euclidean space that are uniformly distributed over several multidimensional concentric hyperspheres. For the 2-dimensional case, such distributions have been previously considered in order to reduce the computational complexity of finding a bound on the channel capacity of fiber-optic c…
In our previous paper [Phys. Rev. E 95, 062122 (2017)] we considered the optical channel modelled by the nonlinear Schrödinger equation with zero dispersion and additive Gaussian noise. We found per-sample channel capacity rof this model. In the present paper we extend per-sample model by introducing the initial signal dependence on time and the output signal detection procedure. The proposed model is a closer approximation of the realistic communication link than the per-sample model where there is no dependence of the initial signal on time. For the proposed model we found the correlators of…
Lower bounds on mutual information (MI) of long-haul optical fiber systems for hard-decision and soft-decision decoding are studied. Ready-to-use expressions to calculate the MI are presented. Extensive numerical simulations are used to quantify how changes in the optical transmitter, receiver, and channel affect the achievable transmission rates of the system. Special emphasis is put to the use of different quadrature amplitude modulation formats, channel spacings, digital back-propagation schemes and probabilistic shaping. The advantages of using MI over the prevailing $Q$-factor as a figure…
Estimating mutual information (MI) from samples is a fundamental problem in statistics, machine learning, and data analysis. Recently it was shown that a popular class of non-parametric MI estimators perform very poorly for strongly dependent variables and have sample complexity that scales exponentially with the true MI. This undesired behavior was attributed to the reliance of those estimators on local uniformity of the underlying (and unknown) probability density function. Here we present a novel semi-parametric estimator of mutual information, where at each sample point, densities are {\em…
In this paper, we suggest a framework to make use of mutual information as a regularization criterion to train Auto-Encoders (AEs). In the proposed framework, AEs are regularized by minimization of the mutual information between input and encoding variables of AEs during the training phase. In order to estimate the entropy of the encoding variables and the mutual information, we propose a non-parametric method. We also give an information theoretic view of Variational AEs (VAEs), which suggests that VAEs can be considered as parametric methods that estimate entropy. Experimental results show t…
MIMO processing techniques in fiber optical communications have been proposed as a promising approach to meet increasing demand for information throughput. In this context, the multiple channels correspond to the multiple modes and/or multiple cores in the fiber. In this paper we characterize the distribution of the mutual information with Gaussian input in a simple channel model for this system. Assuming significant cross talk between cores, negligible backscattering and near-lossless propagation in the fiber, we model the transmission channel as a random complex unitary matrix. The loss in t…
Path-integral approach to mutual information calculation for nonlinear communication channel with small second dispersion at large signal-to-noise power ratio
Nokia（诺基亚）；Aston University（阿斯顿大学）；Delft University of Technology（代尔夫特理工大学）
We present a novel end-to-end autoencoder-based learning for coherent optical communications using a "parallelizable" perturbative channel model. We jointly optimized constellation shaping and nonlinear pre-emphasis achieving mutual information gain of 0.18 bits/sym./pol. simulating 64 GBd dual-polarization single-channel transmission over 30x80 km G.652 SMF link with EDFAs.
In this Letter we analyze the benefit of digital compensation of fiber nonlinearity, where the digital signal processing is divided between the transmitter and receiver. The application of the Gaussian noise model indicates that, where there are two or more spans, it is always beneficial to split the nonlinearity compensation. The theory is verified via numerical simulations, investigating transmission of single channel 50 GBd polarization division multiplexed 256-ary quadrature amplitude modulation over 100 km standard single mode fiber spans, using lumped amplification. For this case, the ad…
We consider the problem of estimating mutual information between dependent data, an important problem in many science and engineering applications. We propose a data-driven, non-parametric estimator of mutual information in this paper. The main novelty of our solution lies in transforming the data to frequency domain to make the problem tractable. We define a novel metric--mutual information in frequency--to detect and quantify the dependence between two random processes across frequency using Cramér's spectral representation. Our solution calculates mutual information as a function of frequen…
Mutual information is a measure of the dependence between random variables that has been used successfully in myriad applications in many fields. Generalized mutual information measures that go beyond classical Shannon mutual information have also received much interest in these applications. We derive the mean squared error convergence rates of kernel density-based plug-in estimators of general mutual information measures between two multidimensional random variables $\mathbf{X}$ and $\mathbf{Y}$ for two cases: 1) $\mathbf{X}$ and $\mathbf{Y}$ are continuous; 2) $\mathbf{X}$ and $\mathbf{Y}$ …
