A machine that learns a task from observations must encounter and process uncertainty and novelty, especially when it is to maintain performance when observing new information and to select the hypothesis that best fits the current observations. In this context, some key questions arise: what and how much information did the observations provide, how much information is required to identify the data-generating process, how many observations remain to get that information, and how does a predictor determine that it has observed novel information? We formalize identifying information to answer these questions and synthesize prior works. Identifying information are bits that verify or falsify a hypothesis as the data-generating process. In this formalization, we prove the information theoretic characteristics of the computation of hypothesis identification and the resulting sample complexity. We define hypothesis identification and sample complexity via the computation of an indicator function over a set of hypotheses, bridging algorithmic and probabilistic information. We detail the sample complexity and its properties for data-generating processes ranging from deterministic processes to ergodic stationary stochastic processes, which connect the notion of identifying information in finite steps with asymptotic statistics and PAC-learning. The indicator function's computation naturally formalizes novel information and its identification from observations with respect to a hypothesis set, which detects a misspecified hypothesis set. We also proved that a computable PAC-Bayes learners' sample complexity distribution is determined by its moments in terms of the prior probability distribution over a fixed finite hypothesis set, and thus an approximation of the sample complexity distribution is always computable within the desired precision that resources allow.
Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Direction
Shenzhen Key Laboratory of Neuropsychiatric Modulation（深圳心理行为调控重点实验室）；Shenzhen-Hong Kong Institute of Brain Science（深圳-香港脑科学研究院）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）
Identifying Drivers of Predictive Aleatoric Uncertainty
Hasso Plattner Institute（霍普夫纳研究所）；Digital Engineering Faculty（数字工程学院）；University of Potsdam（波茨坦大学）；Freie Universität Berlin（柏林自由大学）；Department of Mathematics and Computer Science（数学与计算机科学系）；Hasso Plattner Institute for Digital Health at Mount Sinai（辛克尔医学院霍普夫纳数字健康研究所）；Icahn School of Medicine at Mount Sinai（辛克尔医学院）
VNDUQE: Information-Theoretic Novelty Detection using Deep Variational Information Bottleneck
1 Nondestructive Evaluation Lab, Department of Electrical；Computer Engineering Michigan State University East Lansing, MI Email；2 Department of Electrical
A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
University of Amsterdam（阿姆斯特丹大学）；Sharif University of Technology（谢里夫理工大学）；UC Berkeley（加州大学伯克利分校）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Institute For Research In Fundamental Sciences (IPM)（基础科学研究所）
