Mutual information (MI) is an information-theoretic measure of dependency between two random variables. Several methods to estimate MI, from samples of two random variables with unknown underlying probability distributions have been proposed in the literature. Recent methods realize parametric probability distributions or critic as a neural network to approximate unknown density ratios. The approximated density ratios are used to estimate different variational lower bounds of MI. While these methods provide reliable estimation when the true MI is low, they produce high variance estimates in cases of high MI. We argue that the high variance characteristic is due to the uncontrolled complexity of the critic's hypothesis space. In support of this argument, we use the data-driven Rademacher complexity of the hypothesis space associated with the critic's architecture to analyse generalization error bound of variational lower bound estimates of MI. In the proposed work, we show that it is possible to negate the high variance characteristics of these estimators by constraining the critic's hypothesis space to Reproducing Hilbert Kernel Space (RKHS), which corresponds to a kernel learned using Automated Spectral Kernel Learning (ASKL). By analysing the aforementioned generalization error bounds, we augment the overall optimisation objective with effective regularisation term. We empirically demonstrate the efficacy of this regularization in enforcing proper bias variance tradeoff on four variational lower bounds, namely NWJ, MINE, JS and SMILE.
School of Management, Shenzhen Polytechnic University（深圳职业技术大学管理学院）；School of Data Science, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）数据科学学院）；School of Data Science, Fudan University（复旦大学数据科学学院）
Northeastern University（东北大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Boston, MA, USA（美国马萨诸塞州波士顿（此处按要求不提取，实际应为无，原文本中此条是地址，非机构，正确应只取Northeastern University和Department of Electrical and Computer Engineering，但Department of Electrical and Computer Engineering是系，属于大学内部分类，按要求粒度到大学/学院/公司/研究院级别即可，所以只保留Northeastern University）}]}修正：根据提取规则，机构粒度到大学/学院/公司/研究院级别即可，不要细拆到系，所以正确应为{）
Estimating Kullback Leibler (KL) divergence from samples of two distributions is essential in many machine learning problems. Variational methods using neural network discriminator have been proposed to achieve this task in a scalable manner. However, we noted that most of these methods using neural network discriminators suffer from high fluctuations (variance) in estimates and instability in training. In this paper, we look at this issue from statistical learning theory and function space complexity perspective to understand why this happens and how to solve it. We argue that the cause of th…
University of Florida（佛罗里达大学）；University of Miami（迈阿密大学）；UiT - The Arctic University of Norway（UiT-挪威北极大学）
The matrix-based Renyi's α-order entropy functional was recently introduced using the normalized eigenspectrum of a Hermitian matrix of the projected data in a reproducing kernel Hilbert space (RKHS). However, the current theory in the matrix-based Renyi's α-order entropy functional only defines the entropy of a single variable or mutual information between two random variables. In information theory and machine learning communities, one is also frequently interested in multivariate information quantities, such as the multivariate joint entropy and different interactive quantities among multip…
We describe a novel extension of soft actor-critics for hierarchical Deep Q-Networks (HDQN) architectures using mutual information metric. The proposed extension provides a suitable framework for encouraging explorations in such hierarchical networks. A natural utilization of this framework is an adversarial setting, where meta-controller and controller play minimax over the mutual information objective but cooperate on maximizing expected rewards.
Mutual Information Assisted Ensemble Recommender System for Identifying Critical Risk Factors in Healthcare Prognosis
Bethune College, University of Calcutta（加尔各答大学贝休恩学院）；Jadavpur University（jadavpur大学）；Mahindra University（马恒达大学）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）；King Mongkut’s University of Technology Thonburi（泰国国王科技大学吞武里校区）
Cumulative entropy regularization introduces a regulatory signal to the reinforcement learning (RL) problem that encourages policies with high-entropy actions, which is equivalent to enforcing small deviations from a uniform reference marginal policy. This has been shown to improve exploration and robustness, and it tackles the value overestimation problem. It also leads to a significant performance increase in tabular and high-dimensional settings, as demonstrated via algorithms such as soft Q-learning (SQL) and soft actor-critic (SAC). Cumulative entropy regularization has been extended to o…
Rutgers University（罗格斯大学）；Toyota Technological Institute at Chicago（芝加哥丰田工业大学）
We propose learning discrete structured representations from unlabeled data by maximizing the mutual information between a structured latent variable and a target variable. Calculating mutual information is intractable in this setting. Our key technical contribution is an adversarial objective that can be used to tractably estimate mutual information assuming only the feasibility of cross entropy calculation. We develop a concrete realization of this general formulation with Markov distributions over binary encodings. We report critical and unexpected findings on practical aspects of the objec…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Trustworthy AI is a critical issue in machine learning where, in addition to training a model that is accurate, one must consider both fair and robust training in the presence of data bias and poisoning. However, the existing model fairness techniques mistakenly view poisoned data as an additional bias to be fixed, resulting in severe performance degradation. To address this problem, we propose FR-Train, which holistically performs fair and robust model training. We provide a mutual information-based interpretation of an existing adversarial training-based fairness-only method, and apply this …
University of Rochester（罗切斯特大学）；Amazon Web Services（亚马逊网络服务）
Deep learning algorithms mine knowledge from the training data and thus would likely inherit the dataset's bias information. As a result, the obtained model would generalize poorly and even mislead the decision process in real-life applications. We propose to remove the bias information misused by the target task with a cross-sample adversarial debiasing (CSAD) method. CSAD explicitly extracts target and bias features disentangled from the latent representation generated by a feature extractor and then learns to discover and remove the correlation between the target and bias features. The corr…
University of Guelph（圭尔夫大学）；Vector Institute（向量研究所）；University of Calgary（卡尔加里大学）；Mayo Clinic（梅奥诊所）；University of Waterloo（滑铁卢大学）
The failure of deep neural networks to generalize to out-of-distribution data is a well-known problem and raises concerns about the deployment of trained networks in safety-critical domains such as healthcare, finance and autonomous vehicles. We study a particular kind of distribution shift $\unicode{x2013}$ shortcuts or spurious correlations in the training data. Shortcut learning is often only exposed when models are evaluated on real-world data that does not contain the same spurious correlations, posing a serious dilemma for AI practitioners to properly assess the effectiveness of a traine…
A counter-intuitive property of convolutional neural networks (CNNs) is their inherent susceptibility to adversarial examples, which severely hinders the application of CNNs in security-critical fields. Adversarial examples are similar to original examples but contain malicious perturbations. Adversarial training is a simple and effective defense method to improve the robustness of CNNs to adversarial examples. The mechanisms behind adversarial examples and adversarial training are worth exploring. Therefore, this work investigates similarities and differences between normally trained CNNs (NT…
Variational mutual information (MI) estimators are widely used in unsupervised representation learning methods such as contrastive predictive coding (CPC). A lower bound on MI can be obtained from a multi-class classification problem, where a critic attempts to distinguish a positive sample drawn from the underlying joint distribution from $(m-1)$ negative samples drawn from a suitable proposal distribution. Using this approach, MI estimates are bounded above by $\log m$, and could thus severely underestimate unless $m$ is very large. To overcome this limitation, we introduce a novel estimator…
In recent years, several unsupervised, "contrastive" learning algorithms in vision have been shown to learn representations that perform remarkably well on transfer tasks. We show that this family of algorithms maximizes a lower bound on the mutual information between two or more "views" of an image where typical views come from a composition of image augmentations. Our bound generalizes the InfoNCE objective to support negative sampling from a restricted region of "difficult" contrasts. We find that the choice of negative samples and views are critical to the success of these algorithms. Refo…
Mutual information and task-relevant latent dimensionality
Department of Physics, Initiative for Theory and Modeling of Living Systems, Emory University（物理系、生命系统理论与建模倡议、埃默里大学）；School of Physics, School of Psychology, Georgia Institute of Technology（物理学院、心理学学院、佐治亚理工学院）；Department of Biology, Emory University（生物学系、埃默里大学）
On the Properties and Estimation of Pointwise Mutual Information Profiles
ETH AI Center, ETH Zurich（苏黎世联邦理工学院ETH AI中心）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）；SIB Swiss Institute of Bioinformatics（瑞士SIB生物信息学研究所）；Department of Biosystems Science and Engineering, ETH Zurich（苏黎世联邦理工学院生物系统科学与工程系）；Department of Computer Science, ETH Zurich（苏黎世联邦理工学院计算机科学系）
本文解析多元正态分布的点互信息轮廓，提出Bend and Mix模型族，用蒙特卡洛方法估计轮廓，并用于研究互信息估计器局限、神经评判器行为及异常值影响，最终实现基于模型的贝叶斯互信息估计。
University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Washington（华盛顿大学）；Tata Institute of Fundamental Research（塔塔基础研究院）
Information-theoretic quantities like entropy and mutual information have found numerous uses in machine learning. It is well known that there is a strong connection between these entropic quantities and submodularity since entropy over a set of random variables is submodular. In this paper, we study combinatorial information measures that generalize independence, (conditional) entropy, (conditional) mutual information, and total correlation defined over sets of (not necessarily random) variables. These measures strictly generalize the corresponding entropic measures since they are all paramet…
ETH Zurich（苏黎世联邦理工学院）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Physical systems differring in their microscopic details often display strikingly similar behaviour when probed at macroscopic scales. Those universal properties, largely determining their physical characteristics, are revealed by the powerful renormalization group (RG) procedure, which systematically retains "slow" degrees of freedom and integrates out the rest. However, the important degrees of freedom may be difficult to identify. Here we demonstrate a machine learning algorithm capable of identifying the relevant degrees of freedom and executing RG steps iteratively without any prior knowl…
