AI 中文总结
本文提出重复强化与持续遗忘（RPF）动态框架，从理论上和实验上证明遗忘作为选择机制可产生类似MDL的压缩，提供可控归纳偏置。
AI 中文摘要
先前的研究表明，神经网络在训练过程中会表现出对低复杂度结构（如谱偏置）的隐式偏好、记忆动态以及类似压缩的效应，但对选择性保留的统一动态解释仍不完整。我们提出了带有持续遗忘的重复强化（RPF）动态，这是一个最小框架，其中重复暴露会强化数据中反复出现的模式和结构，而持续遗忘则会衰减已学习的信息。这种观点将遗忘不仅视为一种失败模式，更视为一种选择机制。我们分三个连续层次构建理论。首先，在独立特征模型中，我们推导出暴露选择性生存定律和支持依赖性保留边界，刻画了在遗忘下哪些模式得以持续存在。其次，在共享参数模型中，我们表明遗忘会诱导对协方差模式的谱过滤，保留强支持的共享成分，同时抑制弱支持的成分。第三，在小步长和范数/编码近似下，我们展示了RPF动态如何在数据拟合与存储信息成本之间诱导隐式权衡，产生类似最小描述长度（MDL）的压缩。受控实验为标量记忆和非线性共享网络中的这种强化-遗忘选择机制提供了证据。联合强化和衰减干预会改变条件保留，而匹配的暴露次数揭示了遗忘依赖的强化时机效应以及保留集组成的变化。总之，这些结果表明，重复强化和持续遗忘共同提供了一种超越神经架构和规模的可控归纳偏置来源。
英文摘要
Prior work has shown that neural networks exhibit implicit biases toward low-complexity structure (e.g., spectral bias), memorization dynamics, and compression-like effects during training, but a unified dynamical account of selective retention remains incomplete. We propose Repeated Reinforcement with Persistent Forgetting (RPF) dynamics, a minimal framework in which repeated exposure reinforces patterns and structures that recur in the data, while persistent forgetting attenuates learned information. This view treats forgetting not merely as a failure mode, but as a selection mechanism. We build the theory in three successive layers. First, in an independent-feature model, we derive an exposure-selective survival law and a support-dependent retention boundary characterizing which patterns persist under forgetting. Second, in a shared-parameter model, we show that forgetting induces spectral filtering over covariance modes, preserving strongly supported shared components while suppressing weak ones. Third, under small-step and norm/coding approximations, we show how RPF dynamics induce an implicit trade-off between data fitting and the cost of stored information, yielding Minimum Description Length (MDL)-like compression. Controlled experiments provide evidence for this reinforcement--forgetting selection mechanism in scalar memories and a nonlinear shared network. Joint reinforcement and attenuation interventions shift conditional retention, while matched exposure counts reveal forgetting-dependent effects of reinforcement timing and changes in the composition of the retained set. Together, these results show that repeated reinforcement and persistent forgetting jointly provide a controllable source of inductive bias beyond neural architecture and scale.
Comments19 pages, 3 figures
