从熵到epiplexity：重新思考为计算受限智能的信息
From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
AI总结：
本文重新审视信息理论，提出epiplexity概念，探讨计算受限观察者如何从数据中学习信息，强调结构内容与时间熵的区别，并提供评估方法以指导数据选择与生成。
摘要：
我们能否从数据中学习比生成过程本身更多的内容？仅通过确定性变换是否能构造出新有用的的信息？数据中的可学习内容是否能在不考虑下游任务的情况下评估？在这些问题上，香农信息和柯尔莫哥洛夫复杂性几乎无能为力，部分原因是它们假定观察者具有无限计算能力且不针对有用信息内容。本文识别并举例说明信息理论中的三个看似矛盾：（1）确定性变换无法增加信息；（2）信息与数据顺序无关；（3）似然建模仅仅是分布匹配。为了解决这些结果与现代实践之间的张力，并量化数据的价值，我们引入epiplexity，一种形式化的信息，捕捉计算受限观察者从数据中学习的内容。epiplexity捕捉数据中的结构内容，同时排除时间受限熵，如伪随机数生成器和混沌动力系统所体现的随机不可预测内容。通过这些概念，我们展示了如何通过计算创造信息，如何依赖数据的顺序，以及似然建模如何生成比数据生成过程本身更复杂的程序。我们还提出了实用的估算epiplexity的程序，证明其能捕捉不同数据源之间的差异，跟踪下游性能，并突出数据集干预以改善分布外泛化。与模型选择原则不同，epiplexity为数据选择提供理论基础，指导如何选择、生成或转换数据以供学习系统使用。
英文摘要：
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and do not target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
