时间维度的压缩:可分性、平稳性与容量的相互作用
Compression Through Time: An Interplay of Separability, Stationarity and Capacity
AI总结:
该研究探究循环网络中压缩的时间动态,以离散化LSTM为工具,揭示类别可分性、时间平稳性和网络容量是时间压缩的关键决定因素,明确压缩作为正则化项的利弊。
摘要:
信息论压缩和几何压缩已被广泛研究,作为解释泛化的可能方式,但循环网络中它们的时间动态仍未得到充分探索。我们使用离散化LSTM(其有限嵌入空间可直接估计信息量),观察到在许多数据集的最终时间步存在显著压缩。我们通过三种相互竞争的机制解释该效应:(i)类别可分性允许在最终时间步进行更强的压缩;(ii)共享循环动态将这种压力传播到具有相似激活状态的时间步,使压缩程度依赖于时间平稳性;(iii)有限的表示容量阻碍长期压缩,因为它限制了后续计算可用的信息。我们通过受控合成干预和真实序列任务系统地测试这些机制。结果表明,类别可分性、时间平稳性和网络容量是时间压缩的关键决定因素,且压缩可作为正则化项——当存在过剩容量时可能有益,但当它占用任务所需内存时会损害泛化能力。
英文摘要:
Information-theoretic and geometric compression have been widely studied as possible explanations for generalization. Yet their temporal dynamics in recurrent networks remain largely unexplored. Using discretized LSTMs, whose finite embedding space enables direct estimates of information quantities, we observe pronounced compression across the final timesteps of many datasets. We explain this effect through three competing mechanisms: (i) class separability permits stronger compression at the final timestep; (ii) shared recurrent dynamics propagate this pressure across timesteps with similar activation states, making its extent depend on temporal stationarity; and (iii) limited representational capacity opposes prolonged compression because it restricts information available for subsequent computation. We systematically test these mechanisms through controlled synthetic interventions and real-world sequence tasks. Our results identify class separability, temporal stationarity, and network capacity as key determinants of temporal compression, and suggest that compression acts as a regularizer-potentially beneficial when excess capacity is available, but harmful to generalization when it consumes memory needed for the task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。