超越参数空间的PAC-贝叶斯：行为等价性、Z-信息与精确复杂度分解

中文摘要 AI 辅助

PAC-贝叶斯理论通过控制所选假设表示的后验分布与先验分布之间的KL散度，提供泛化保证。然而，预测风险仅取决于假设诱导的预测行为，而非实现该行为的特定内部实现。在过参数化系统中，许多不同的配置会诱导出相同的预测行为，但经典PAC-贝叶斯KL散度无法区分对预测行为的不确定性与行为等价实现之间的差异。我们证明，这种区分会诱导出经典PAC-贝叶斯复杂度的精确结构分解。我们通过可测行为映射形式化行为等价性，并利用测度 disintegration 将配置空间上的概率测度分解为预测行为的分布和行为纤维上的条件分布。这将经典PAC-贝叶斯KL散度精确分解为行为选择项和由纤维内期望条件KL给出的实现级项。我们将Z-信息定义为该实现级贡献的负值，即KL散度与仅关于预测行为的不确定性复杂度之间的精确差距。我们进一步证明，行为选择项具有精确的变分表征：它是诱导相同预测行为分布的所有后验中最小的KL散度，由规范的纤维对称化代表达到。最后，我们证明对称性、行为保持方向、纤维几何以及纤维保持扰动下的不变性均自然源于同一行为映射结构。这些结果共同确定预测行为是PAC-贝叶斯复杂度的自然对象。

英文摘要

PAC-Bayes theory provides generalization guarantees by controlling the Kullback--Leibler (KL) divergence between posterior and prior distributions over a chosen hypothesis representation. However, predictive risk depends only on the predictive behavior induced by a hypothesis, not on the particular internal realization that implements that behavior. In over-parameterized systems, many distinct configurations induce identical predictive behavior, yet the classical PAC-Bayes KL divergence does not distinguish uncertainty over predictive behavior from variation among behaviorally equivalent realizations. We show that this distinction induces an exact structural decomposition of classical PAC-Bayes complexity. We formalize behavioral equivalence through a measurable behavior map and use measure disintegration to decompose probability measures on the configuration space into a distribution over predictive behaviors and conditional distributions over behavioral fibers. This yields an exact decomposition of the classical PAC-Bayes KL divergence into a behavior-selection term and a realization-level term given by an expected conditional KL within fibers. We define Z-information as the negative of this realization-level contribution: the exact gap between the KL divergence and the complexity of uncertainty over predictive behavior alone. We further show that the behavior-selection term admits an exact variational characterization: it is the minimum KL divergence among all posteriors inducing the same distribution over predictive behaviors, attained by a canonical fiber-symmetrized representative. Finally, we show that symmetry, behavior-preserving directions, fiber geometry, and invariance under fiber-preserving perturbations arise naturally from the same behavior-map structure. Together, these results identify predictive behavior as the natural object of PAC-Bayes complexity.

发表机构

• The Pennsylvania State University（宾夕法尼亚州立大学）
• Artificial Intelligence Research Laboratory（人工智能研究实验室）

机构由 AI 辅助整理，请以论文原文为准。