端到端训练通过层级角色分化诱导信息瓶颈：与逐层训练的对比分析
End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Training
AI总结：
本文通过与逐层训练对比，发现端到端训练因能高效传播信息并促使层级角色分化，使最终表示遵循信息瓶颈原则，揭示了层间协同交互对深度学习信息瓶颈分析的重要性。
摘要：
端到端（E2E）训练通过误差反向传播优化整个模型，从根本上支撑了深度学习的进步。尽管E2E训练具有高性能，但它面临内存消耗、并行计算以及与实际大脑功能存在差异等问题。为了克服这些困难，人们提出了各种替代方法；然而，目前还没有一种方法能匹敌E2E训练的性能，因此在实用性上有所欠缺。此外，对于性能差距之外的训练模型属性差异，目前还没有深入的理解。在本文中，我们通过与逐层训练（一种局部设定误差的非E2E方法）进行比较，重新审视了E2E训练表现出优越性能的原因。基于E2E训练在传播输入信息方面具有优势这一观察，我们利用Hilbert-Schmidt独立性准则（HSIC）分析了中间表示的信息平面动态。我们对归一化HSIC值的分析结果表明，除了高效的信息传播外，E2E训练还具有在不同层间展现不同信息动态的能力。此外，我们展示了这种层级角色分化使得最终表示遵循信息瓶颈原则。这表明，在分析深度学习的信息瓶颈时，需要考虑层与层之间的协同交互，而不仅仅是最终的输出层。
英文摘要：
End-to-end (E2E) training, optimizing the entire model through error backpropagation, fundamentally supports the advancements of deep learning. Despite its high performance, E2E training faces the problems of memory consumption, parallel computing, and discrepancy with the functionalities of the actual brain. Various alternative methods have been proposed to overcome these difficulties; however, no one can yet match the performance of E2E training, thereby falling short in practicality. Furthermore, there is no deep understanding regarding differences in the trained model properties beyond the performance gap. In this paper, we reconsider why E2E training demonstrates a superior performance through a comparison with layer-wise training, a non-E2E method that locally sets errors. On the basis of the observation that E2E training has an advantage in propagating input information, we analyze the information plane dynamics of intermediate representations based on the Hilbert-Schmidt independence criterion (HSIC). The results of our normalized HSIC value analysis reveal the E2E training ability to exhibit different information dynamics across layers, in addition to efficient information propagation. Furthermore, we show that this layer-role differentiation leads to the final representation following the information bottleneck principle. It suggests the need to consider the cooperative interactions between layers, not just the final layer when analyzing the information bottleneck of deep learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
