Information Theory in Density Destructors

Obfuscation via Information Density Estimation

Identifying features that leak information about sensitive attributes is a key challenge in the design of information obfuscation mechanisms. In this paper, we propose a framework to identify information-leaking features via information density estimation. Here, features whose information densities exceed a pre-defined threshold are deemed information-leaking features. Once these features are identified, we sequentially pass them through a targeted obfuscation mechanism with a provable leakage guarantee in terms of $\mathsf{E}_γ$-divergence. The core of this mechanism relies on a data-driven e…

Generalization Bounds via Information Density and Conditional Information Density

We present a general approach, based on exponential inequalities, to derive bounds on the generalization error of randomized learning algorithms. Using this approach, we provide bounds on the average generalization error as well as bounds on its tail probability, for both the PAC-Bayesian and single-draw scenarios. Specifically, for the case of sub-Gaussian loss functions, we obtain novel bounds that depend on the information density between the training data and the output hypothesis. When suitably weakened, these bounds recover many of the information-theoretic bounds available in the litera…

Inverting Supervised Representations with Autoregressive Neural Density Models

We present a method for feature interpretation that makes use of recent advances in autoregressive density estimation models to invert model representations. We train generative inversion models to express a distribution over input features conditioned on intermediate model representations. Insights into the invariances learned by supervised models can be gained by viewing samples from these inversion models. In addition, we can use these inversion models to estimate the mutual information between a model's inputs and its intermediate representations, thus quantifying the amount of information…

Generalization Error Bounds via $m$th Central Moments of the Information Density

We present a general approach to deriving bounds on the generalization error of randomized learning algorithms. Our approach can be used to obtain bounds on the average generalization error as well as bounds on its tail probabilities, both for the case in which a new hypothesis is randomly generated every time the algorithm is used - as often assumed in the probably approximately correct (PAC)-Bayesian literature - and in the single-draw case, where the hypothesis is extracted only once. For this last scenario, we present a novel bound that is explicit in the central moments of the information…

通过密度近似估计期望信息增益：样本分配与维度约简

Expected information gain estimation via density approximations: Sample allocation and dimension reduction

本文提出了一种基于密度近似的EIG估计方法，通过优化样本分配和维度约简，提高了高维情况下EIG估计的效率和准确性。

期权隐含信息的浅层表示

Shallow Representation of Option Implied Information

本文提出通过浅层神经网络构建期权隐含信息的表示方法，揭示隐含密度与隐含波动率的联系，并通过实验表明单层网络能有效近似两者。