发表机构
DTU Compute; Technical University of Denmark; Faculty of Health Sciences and Medicine; University of Lucerne（丹麦技术大学计算系; 丹麦技术大学; 健康科学与医学学院; 卢塞恩大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文揭示信息论不确定性分解中AU和EU存在不可行区域，其边界为AU≤log(2)/N，解释认知坍缩并指出增加集成规模可缓解，且低AU时AU与EU耦合。
AI 中文摘要
机器学习中的不确定性估计通常利用标准信息论框架将不确定性分解为偶然不确定性（AU）和认知不确定性（EU）。然而，在实践中会出现两个关键问题：纠缠（AU和EU高度相关）和认知坍缩（EU量值随模型容量增加而缩小）。我们在功能层面分析该框架，发现所假设的AU、EU范围中相当一部分在有限设置下不可行，且无法通过任何类别概率实现。我们刻画了这一不可行区域随类别数和蒙特卡洛样本数$N$（例如，由$N$个成员组成的集成）的缩放规律，揭示其边界为$\text{AU} \leq \frac{\text{log}(2)}{N}$。关键的是，不可行区域的边界有助于解释认知坍缩：当模型置信度较高时，$\text{AU} > \text{EU}$由这一基本结构限制保证。我们的发现表明，增加集成规模通过减小不可行区域来缓解认知坍缩。最后，我们提醒在低AU区域中不应将AU和EU视为独立量，因为我们证明当$\text{AU} \leq \frac{\text{log}(2)}{N}$时它们相互耦合。
英文摘要
Uncertainty estimation in machine learning typically decomposes uncertainty into aleatoric uncertainty (AU) and epistemic uncertainty (EU) using the standard information-theoretic framework. However, in practice, two critical issues arise: entanglement (AU and EU are highly correlated) and epistemic collapse (EU magnitude shrinks with increasing model capacity). We analyze this framework on a functional level and discover that significant portions of the assumed AU, EU range are infeasible in finite settings, and cannot be attained with any class probabilities. We characterize how this infeasible region scales with the number of classes and Monte Carlo samples $N$ (e.g., from ensembles with $N$ members), revealing it is bounded by $\text{AU} \leq \log(2)/N$. Crucially, the infeasible region's boundary helps explain epistemic collapse: when model confidence is high, $\text{AU} > \text{EU}$ is guaranteed by this fundamental structural limitation. Our findings show that increasing ensemble size mitigates epistemic collapse by reducing the infeasible area. Lastly, we caution against interpreting AU and EU as independent quantities in low AU regimes, since we show they are coupled when $\text{AU} \leq \log(2)/N$.
