On the Estimation of Information Measures of Continuous Distributions
Institute of Telecommunications, TU Wien（维也纳工业大学电信研究所）；Université Paris-Saclay, CNRS, CentraleSupélec, Laboratoire des Signaux et Systèmes（巴黎萨克雷大学、法国国家科学研究中心、中央理工-高等电力学院、信号与系统实验室）；Montreal Institute for Learning Algorithms (Mila), Université de Montréal（蒙特利尔学习算法研究所（Mila）、蒙特利尔大学）；Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）
Georg Pichler, Pablo Piantanida, Günther Koliander
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space, computed from a finite number of samples, when the probability density function belongs to a predetermined convex family $\mathcal{P}$. First, estimating differential entropy to any accuracy is shown to be infeasible if the differential entropy of densities in $\mathcal{P}$ is unbounded, clearly showing the necessity of additional assumptions. Subsequently, we investigate sufficient conditions that enable confidence bounds for the estimation of differential entropy. In particular, we provide confidence bounds for simple histogram based estimation of differential entropy from a fixed number of samples, assuming that the probability density function is Lipschitz continuous with known Lipschitz constant and known, bounded support. Our focus is on differential entropy, but we provide examples that show that similar results hold for mutual information and relative entropy as well.
We consider the problem of asymptotically efficient estimation of drift parameters of the ergodic fractional Ornstein-Uhlenbeck process under continuous observations when the Hurst parameter $H<1/2$ and the mean of its stationary distribution is not equal to zero. In this paper, we derive asymptotically efficient rates and variances of estimators of drift parameters and prove an asymptotic efficiency of a maximum likelihood estimator of drift parameters.
We consider the nonparametric estimation problem of continuous probability distribution functions. For the integrated mean square error we provide the statistic corresponding to the best invariant estimator proposed by Aggarwal (1955) and Ferguson (1967). The table of critical values is computed and a numerical power comparison of the statistic with the traditional Cramér-von Mises statistic is done for several representative distributions.
We prove strong consistency and asymptotic normality of least squares estimators for the subcritical Heston model based on continuous time observations. We also present some numerical illustrations of our results.
We study asymptotic properties of conditional least squares estimators for the drift parameters of two-factor affine diffusions based on continuous time observations. We distinguish three cases: subcritical, critical and supercritical. For all the drift parameters, in the subcritical and supercritical cases, asymptotic normality and asymptotic mixed normality is proved, while in the critical case, non-standard asymptotic behavior is described.
In this paper we study the problem of statistical inference for a continuous-time moving average Lévy process of the form $$Z_{t} = \int_{\mathbb{R}}\mathcal{K}(t-s)\, dL_{s},\quad t\in\mathbb{R}$$ with a deterministic kernel (\K\) and a L{é}vy process (L\). Especially the estimation of the Lévy measure (ν\) of $L$ from low-frequency observations of the process $Z$ is considered. We construct a consistent estimator, derive its convergence rates and illustrate its performance by a numerical example. On the technical level, the main challenge is to establish a kind of exponential mixing for cont…
We refer by threshold Ornstein-Uhlenbeck to a continuous-time threshold autoregressive process. It follows the Ornstein-Uhlenbeck dynamics when above or below a fixed level, yet at this level (threshold) its coefficients can be discontinuous. We discuss (quasi)-maximum likelihood estimation of the drift parameters, both assuming continuous and discrete time observations. In the ergodic case, we derive consistency and speed of convergence of these estimators in long time and high frequency. Based on these results, we develop a test for the presence of a threshold in the dynamics. Finally, we ap…
We consider a stochastic differential equation of the form $dr_t = (a - b r_t) dt + σr_t^βdW_t$, where $a$, $b$ and $σ$ are positive constants, $β\in(\frac12,1)$. We study the estimation of an unknown drift parameter $(a,b)$ by continuous observations of a sample path $\{r_t, t \in [0,T]\}$. We prove the strong consistency and asymptotic normality of the maximum likelihood estimator. We propose another strongly consistent estimator, which generalizes an estimator proposed in Dehtiar et al. (2021) for $β=\frac12$. The identification of the diffusion parameters $σ$ and $β$ is discussed as well.
Asymptotics of maximum likelihood estimation for $α$-stable law are analytically investigated with a continuous parameterization. The consistency and asymptotic normality are shown on the interior of the whole parameter space. Although these asymptotics have been provided with Zolotarev's $(B)$ parameterization, there are several gaps between. Especially in the latter, the density, so that scores and their derivatives are discontinuous at $α=1$ for $β\neq 0$ and usual asymptotics are impossible. This is considerable inconvenience for applications. By showing that these quantities are smooth in…
Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the k nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal.
We introduce the notion of continuous invertibility on a compact set for volatility models driven by a Stochastic Recurrence Equation (SRE). We prove the strong consistency of the Quasi Maximum Likelihood Estimator (QMLE) when the optimization procedure is done on a continuously invertible domain. This approach gives for the first time the strong consistency of the QMLE used by Nelson in \cite{nelson:1991} for the EGARCH(1,1) model under explicit but non observable conditions. In practice, we propose to stabilize the QMLE by constraining the optimization procedure to an empirical continuously …
In this paper, we are interested in nonparametric kernel estimation of a generalized regression function, including conditional cumulative distribution and conditional quantile functions, based on an incomplete sample $(X_t, Y_t, ζ_t)_{t\in \mathbb{ R}^+}$ copies of a continuous-time stationary ergodic process $(X, Y, ζ)$. The predictor $X$ is valued in some infinite-dimensional space, whereas the real-valued process $Y$ is observed when $ζ= 1$ and missing whenever $ζ= 0$. Pointwise and uniform consistency (with rates) of these estimators as well as a central limit theorem are established. Con…
In this paper, local linear estimators are adapted for the unknown infinitesimal coefficients associated with continuous-time asset return model with jumps, which can correct the bias automatically due to their simple bias representation. The integrated diffusion models with jumps, especially infinite activity jumps are mainly investigated. In addition, under mild conditions, the weak consistency and asymptotic normality is provided through the conditional Lindeberg theorem. Furthermore, our method presents advantages in bias correction through simulation whether jumps belong to the finite act…
We identify the average dose-response function (ADRF) for a continuously valued error-contaminated treatment by a weighted conditional expectation. We then estimate the weights nonparametrically by maximising a local generalised empirical likelihood subject to an expanding set of conditional moment equations incorporated into the deconvolution kernels. Thereafter, we construct a deconvolution kernel estimator of ADRF. We derive the asymptotic bias and variance of our ADRF estimator and provide its asymptotic linear expansion, which helps conduct statistical inference. To select our smoothing p…
Targeted maximum likelihood estimation is a general methodology combining flexible ensemble learning and semiparametric efficiency theory in a two-step procedure for estimation of causal parameters. Proposed targeted maximum likelihood procedures for survival and competing risks analysis have so far focused on events taken values in discrete time. We here present a targeted maximum likelihood estimation procedure for event times that take values in R+. We focuson the estimation of intervention-specific mean outcomes with stochastic interventions on a time-fixed treatment. For data-adaptive est…
In this paper, we investigate quasi-maximum likelihood (QML) estimation for the parameters of a cointegrated solution of a continuous-time linear state space model observed at discrete time points. The class of cointegrated solutions of continuous-time linear state space models is equivalent to the class of cointegrated continuous-time ARMA (MCARMA) processes. As a start, some pseudo-innovations are constructed to be able to define a QML-function. Moreover, the parameter vector is divided appropriately in long-run and short-run parameters using a representation for cointegrated solutions of co…
Let X be a second order random process indexed by a compact interval [0,T]. Assume that n independent realizations of X are observed on a fixed grid of p time points. Under mild regularity assumptions on the sample paths of X, we show the asymptotic normality of suitable nonparametric estimators of the trend function mu = EX in the space C([0,T]) as n, p go to infinity and, using Gaussian process theory, we derive approximate simultaneous confidence bands for mu.
In this paper we present a novel method for estimating the parameters of a parametric diffusion processes. Our approach is based on a closed-form Maximum Likelihood estimator for an approximating Continuous Time Markov Chain (CTMC) of the diffusion process. Unlike typical time discretization approaches, such as psuedo-likelihood approximations with Shoji-Ozaki or Kessler's method, the CTMC approximation introduces no time-discretization error during parameter estimation, and is thus well-suited for typical econometric situations with infrequently sampled data. Due to the structure of the CTMC,…
In practice, data often contain discrete variables. But most of the popular nonparametric estimation methods have been developed in a purely continuous framework. A common trick among practitioners is to make discrete variables continuous by adding a small amount of noise. We show that this approach is justified if the noise distribution belongs to a certain class. In this case, any estimator developed in a purely continuous framework extends naturally to the mixed data setting. Estimators defined that way will be called continuous convolution estimators. They are extremely easy to implement a…
Continuous-time state-space models (SSMs) are flexible tools for analysing irregularly sampled sequential observations that are driven by an underlying state process. Corresponding applications typically involve restrictive assumptions concerning linearity and Gaussianity to facilitate inference on the model parameters via the Kalman filter. In this contribution, we provide a general continuous-time SSM framework, allowing both the observation and the state process to be non-linear and non-Gaussian. Statistical inference is carried out by maximum approximate likelihood estimation, where multip…
Semi-continuous data comes from a distribution that is a mixture of the point mass at zero and a continuous distribution with support on the positive real line. A clear example is the daily rainfall data. In this paper, we present a novel algorithm to estimate the density function for semi-continuous data using the principle of maximum entropy. Unlike existing methods in the literature, our algorithm needs only the sample values of the constraint functions in the entropy maximization problem and does not need the entire sample. Using simulations, we show that the estimate of the entropy produc…
In marketing we are often confronted with a continuous stream of responses to marketing messages. Such streaming data provide invaluable information regarding message effectiveness and segmentation. However, streaming data are hard to analyze using conventional methods: their high volume and the fact that they are continuously augmented means that it takes considerable time to analyze them. We propose a method for estimating a finite mixture of logistic regression models which can be used to cluster customers based on a continuous stream of responses. This method, which we coin oFMLR, allows s…
When modeling a probability distribution with a Bayesian network, we are faced with the problem of how to handle continuous variables. Most previous work has either solved the problem by discretizing, or assumed that the data are generated by a single Gaussian. In this paper we abandon the normality assumption and instead use statistical methods for nonparametric density estimation. For a naive Bayesian classifier, we present experimental results on a variety of natural and artificial domains, comparing two methods of density estimation: assuming normality and modeling each conditional distrib…
Two fundamental problems in unsupervised learning are efficient inference for latent-variable models and robust density estimation based on large amounts of unlabeled data. Algorithms for the two tasks, such as normalizing flows and generative adversarial networks (GANs), are often developed independently. In this paper, we propose the concept of {\em continuous-time flows} (CTFs), a family of diffusion-based methods that are able to asymptotically approach a target distribution. Distinct from normalizing flows and GANs, CTFs can be adopted to achieve the above two goals in one framework, with…
We propose an inferential approach for maximum likelihood estimation of the hidden Markov models for continuous responses. We extend to the case of longitudinal observations the finite mixture model of multivariate Gaussian distributions with Missing At Random (MAR) outcomes, also accounting for possible dropout. The resulting hidden Markov model accounts for different types of missing pattern: (i) partially missing outcomes at a given time occasion; (ii) completely missing outcomes at a given time occasion (intermittent pattern); (iii) dropout before the end of the period of observation (mono…
Weighted estimators are commonly used for estimating exposure effects in observational settings to establish causal relations. These estimators have a long history of development when the exposure of interest is binary and where the weights are typically functions of an estimated propensity score. Recent developments in optimization-based estimators for constructing weights in binary exposure settings, such as those based on entropy balancing, have shown more promise in estimating treatment effects than those methods that focus on the direct estimation of the propensity score using likelihood-…
Kernel smoothing is a highly flexible and popular approach for estimation of probability density and intensity functions of continuous spatial data. In this role it also forms an integral part of estimation of functionals such as the density-ratio or "relative risk" surface. Originally developed with the epidemiological motivation of examining fluctuations in disease risk based on samples of cases and controls collected over a given geographical region, such functions have also been successfully employed across a diverse range of disciplines where a relative comparison of spatial density funct…
Structural failure time models are causal models for estimating the effect of time-varying treatments on a survival outcome. G-estimation and artificial censoring have been proposed to estimate the model parameters in the presence of time-dependent confounding and administrative censoring. However, most of existing methods require manually preprocessing data into regularly spaced data, which may invalidate the subsequent causal analysis. Moreover, the computation and inference are challenging due to the non-smoothness of artificial censoring. We propose a class of continuous-time structural fa…
This paper proposes a distributed alternating mixed discrete-continuous (DAMDC) algorithm to approach the oracle algorithm based on the diffusion strategy for parameter and spectrum estimation over sensor networks. A least mean squares (LMS) type algorithm that obtains the oracle matrix adaptively is developed and compared with the existing sparsity-aware and conventional algorithms. The proposed algorithm exhibits improved performance in terms of mean square deviation and power spectrum estimation accuracy. Numerical results show that the DAMDC algorithm achieves excellent performance.
The factor graph approach to discrete-time linear Gaussian state space models is well developed. The paper extends this approach to continuous-time linear systems/filters that are driven by white Gaussian noise. By Gaussian message passing, we then obtain MAP/MMSE/LMMSE estimates of the input signal, or of the state, or of the output signal from noisy observations of the output signal. These estimates may be obtained with arbitrary temporal resolution. The proposed input signal estimation does not seem to have appeared in the prior Kalman filtering literature.
This work develops a new method for estimating and optimizing the directed information rate between two jointly stationary and ergodic stochastic processes. Building upon recent advances in machine learning, we propose a recurrent neural network (RNN)-based estimator which is optimized via gradient ascent over the RNN parameters. The estimator does not require prior knowledge of the underlying joint and marginal distributions. The estimator is also readily optimized over continuous input processes realized by a deep generative model. We prove consistency of the proposed estimation and optimiza…
A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over "space" partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimat…
W. M. Keck Science Department of Pitzer, Scripps, and Claremont McKenna College（皮泽学院、斯克里普斯学院和克莱蒙特·麦肯纳学院W.M.凯克科学系）；University of California at Davis（加州大学戴维斯分校）
Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide new methods for inferring, predicting, and estimating them. The methods rely on an extension of Bayesian structural inference that takes advantage of neural network's universal approximation power. Based on experiments with complex synthetic data, the methods are competitive with the state-of-the-art for prediction and entropy-rate estimation.
Optimal design of distributed decision policies can be a difficult task, illustrated by the famous Witsenhausen counterexample. In this paper we characterize the optimal control designs for the vector-valued setting assuming that it results in an internal state that can be described by a continuous random variable which has a probability density function. More specifically, we provide a genie-aided outer bound that relies on our previous results for empirical coordination problems. This solution turns out to be not optimal in general, since it consists of a time-sharing strategy between two li…
