首页/论文/1902.00580

arXiv 1902.005802019-02cs.ITmath.IT

On the Bias of Directed Information Estimators

Gabriel Schamberg, Todd P. Coleman

英文摘要：

When estimating the directed information between two jointly stationary Markov processes, it is typically assumed that the recipient of the directed information is itself Markov of the same order as the joint process. While this assumption is often made explicit in the presentation of such estimators, a characterization of when we can expect the assumption to hold is lacking. Using the concept of d-separation from Bayesian networks, we present sufficient conditions for which this assumption holds. We further show that the set of parameters for which the condition is not also necessary has Lebesgue measure zero. Given the strictness of these conditions, we introduce a notion of partial directed information, which can be used to bound the bias of directed information estimates when the directed information recipient is not itself Markov. Lastly we estimate this bound on simulations in a variety of settings to assess the extent to which the bias should be cause for concern.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Universal Estimation of Directed Information

Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments. The first one is a Shannon--McMillan--Breiman type estimator, similar to those used by Verdú (2005) and Cai, Kulkarni, and Verdú (2006) for estimation of other information measures. We show the almost sure and $L_1$ convergence properties of the estimator for any underlying universal probability assignment. The other three estimators map universal probability assignments to different functionals, each exhibiting relati…

2013-05-30cs.ITmath.IT

$k$-NN Estimation of Directed Information

This report studies data-driven estimation of the directed information (DI) measure between two{em discrete-time and continuous-amplitude} random process, based on the $k$-nearest-neighbors ($k$-NN) estimation framework. Detailed derivations of two $k$-NN estimators are provided. The two estimators differ in the metric based on which the nearest-neighbors are found. To facilitate the estimation of the DI measure, it is assumed that the observed sequences are (jointly) Markovian of order $m$. As $m$ is generally not known, a data-driven method (that is also based on the $k$-NN principle) for es…

2017-11-22cs.ITmath.IT

Neural Estimation and Optimization of Directed Information over Continuous Spaces

This work develops a new method for estimating and optimizing the directed information rate between two jointly stationary and ergodic stochastic processes. Building upon recent advances in machine learning, we propose a recurrent neural network (RNN)-based estimator which is optimized via gradient ascent over the RNN parameters. The estimator does not require prior knowledge of the underlying joint and marginal distributions. The estimator is also readily optimized over continuous input processes realized by a deep generative model. We prove consistency of the proposed estimation and optimiza…

2022-03-28cs.ITmath.IT

SALZA: Soft algorithmic complexity estimates for clustering and causality inference

A complete set of practical estimators for the conditional, simple and joint algorihmic complexities is presented, from which a semi-metric is derived. Also, new directed information estimators are proposed that are applied to causality inference on Directed Acyclic Graphs. The performances of these estimators are investigated and shown to compare well with respect to the state-of-the-art Normalized Compression Distance (NCD).

2016-07-18cs.ITmath.IT

Source Enumeration via RMT Estimator Based on Linear Shrinkage Estimation of Noise Eigenvalues Using Relatively Few Samples

Estimating the number of signals embedded in noise is a fundamental problem in array signal processing. The classic RMT estimator based on random matrix theory (RMT) tends to under-estimate the number of signals as it does not consider the non-negligible bias term among eigenvalues for finite sample size. Moreover, the RMT estimator suffers from uncertainty in noise variance estimation problem. In order to overcome these problems, we firstly derive a more accurate expression for the distribution of the sample eigenvalues and the bias term among eigenvalues by utilizing the linear shrinkage (LS…

2020-10-27cs.ITmath.IT

On Detection-Directed Estimation Approach for Noisy Compressive Sensing

In this paper, we investigate a Bayesian sparse reconstruction algorithm called compressive sensing via Bayesian support detection (CS-BSD). This algorithm is quite robust against measurement noise and achieves the performance of a minimum mean square error (MMSE) estimator that has support knowledge beyond a certain SNR threshold. The key idea behind CS-BSD is that reconstruction takes a detection-directed estimation structure consisting of two parts: support detection and signal value estimation. Belief propagation (BP) and a Bayesian hypothesis test perform support detection, and an MMSE es…

2012-05-14cs.ITmath.IT

更多相关论文收起更多论文

EEG Spatial Decoding and Classification with Logit Shrinkage Regularized Directed Information Assessment (L-SODA)

There is an increasing interest in studying the neural interaction mechanisms behind patterns of cognitive brain activity. This paper proposes a new approach to infer such interaction mechanisms from electroencephalographic (EEG) data using a new estimator of directed information (DI) called logit shrinkage optimized directed information assessment (L-SODA). Unlike previous directed information measures applied to neural decoding, L-SODA uses shrinkage regularization on multinomial logistic regression to deal with the high dimensionality of multi-channel EEG signals and the small sizes of many…

2014-04-01cs.ITmath.IT

Capacity of Continuous Channels with Memory via Directed Information Neural Estimator

Ben Gurion University（本·古里安大学）；Cornell University（康奈尔大学）

Calculating the capacity (with or without feedback) of channels with memory and continuous alphabets is a challenging task. It requires optimizing the directed information (DI) rate over all channel input distributions. The objective is a multi-letter expression, whose analytic solution is only known for a few specific cases. When no analytic solution is present or the channel model is unknown, there is no unified framework for calculating or even approximating capacity. This work proposes a novel capacity estimation algorithm that treats the channel as a `black-box', both when feedback is or …

2020-05-16cs.ITcs.LGmath.IT

Estimating the Directed Information and Testing for Causality

The problem of estimating the directed information rate between two discrete processes $\{X_n\}$ and $\{Y_n\}$ via the plug-in (or maximum-likelihood) estimator is considered. When the joint process $\{(X_n,Y_n)\}$ is a Markov chain of a given memory length, the plug-in estimator is shown to be asymptotically Gaussian and to converge at the optimal rate $O(1/\sqrt{n})$ under appropriate conditions; this is the first estimator that has been shown to achieve this rate. An important connection is drawn between the problem of estimating the directed information rate and that of performing a hypoth…

2016-03-31cs.ITmath.ITmath.ST

Identifying Seizure Onset Zone from the Causal Connectivity Inferred Using Directed Information

In this paper, we developed a model-based and a data-driven estimator for directed information (DI) to infer the causal connectivity graph between electrocorticographic (ECoG) signals recorded from brain and to identify the seizure onset zone (SOZ) in epileptic patients. Directed information, an information theoretic quantity, is a general metric to infer causal connectivity between time-series and is not restricted to a particular class of models unlike the popular metrics based on Granger causality or transfer entropy. The proposed estimators are shown to be almost surely convergent. Causal …

2016-08-16cs.ITmath.ITq-bio.NC

A note on the unbiased estimation of mutual information

Estimators for mutual information are typically biased. However, in the case of the Kozachenko-Leonenko estimator for metric spaces, a type of nearest neighbour estimator, it is possible to calculate the bias explicitly.

2021-05-18cs.ITmath.IT

关于高斯序列因果条件定向信息率的非渐近误差界

Non-Asymptotic Error Bounds for Causally Conditioned Directed Information Rates of Gaussian Sequences

University of Minnesota, Twin Cities（明尼苏达大学双城分校）

本文研究了高斯序列的因果条件定向信息率，提出基于最优预测的显式公式，并给出误差界为O(N^{-1/2}log(N))的估计器。

2026-05-15cs.ITmath.ITmath.ST

Estimating Conditional Transfer Entropy in Time Series using Mutual Information and Non-linear Prediction

We propose a new estimator to measure directed dependencies in time series. The dimensionality of data is first reduced using a new non-uniform embedding technique, where the variables are ranked according to a weighted sum of the amount of new information and improvement of the prediction accuracy provided by the variables. Then, using a greedy approach, the most informative subsets are selected in an iterative way. The algorithm terminates, when the highest ranked variable is not able to significantly improve the accuracy of the prediction as compared to that obtained using the existing sele…

2020-11-27cs.ITmath.ITstat.ME

RMT Estimator with Adaptive Decision Criteria for Estimating the Number of Signals Based on Random Matrix Theory

Estimating the number of signals embedded in noise is a fundamental problem in signal processing. As a classic estimator based on random matrix theory (RMT), the RMT estimator estimates the number of signals via sequentially testing the likelihood of an eigenvalue as arising from a signal or noise for a given over-detection probability. However, it tends to under-estimate the number of signals as weak signal eigenvalues may be immersed in the non-negligible bias term among eigenvalues for finite sample size. In order to solve this problem, we propose an RMT estimator with adaptive decision cri…

2019-11-26cs.ITmath.ITstat.ME

Bias-corrected estimator for intrinsic dimension and differential entropy--a visual multiscale approach

Intrinsic dimension and differential entropy estimators are studied in this paper, including their systematic bias. A pragmatic approach for joint estimation and bias correction of these two fundamental measures is proposed. Shared steps on both estimators are highlighted, along with their useful consequences to data analysis. It is shown that both estimators can be complementary parts of a single approach, and that the simultaneous estimation of differential entropy and intrinsic dimension give meaning to each other, where estimates at different observation scales convey different perspective…

2020-04-30cs.CVcs.ITcs.LG

Bias Correction with Jackknife, Bootstrap, and Taylor Series

University of California, Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）

We analyze bias correction methods using jackknife, bootstrap, and Taylor series. We focus on the binomial model, and consider the problem of bias correction for estimating $f(p)$, where $f \in C[0,1]$ is arbitrary. We characterize the supremum norm of the bias of general jackknife and bootstrap estimators for any continuous functions, and demonstrate the in delete-$d$ jackknife, different values of $d$ may lead to drastically different behaviors in jackknife. We show that in the binomial model, iterating the bootstrap bias correction infinitely many times may lead to divergence of bias and va…

2020-06-16cs.ITcs.LGmath.IT