压缩还是不压缩：自监督学习与信息理论综述
To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review
AI总结：
本文从信息理论视角审视自监督学习方法，引入统一框架涵盖自监督信息理论学习问题，探讨当代方法论、研究方向及信息理论量的经验评估，提供信息理论、自监督学习与深度神经网络交叉领域的详尽综述。
摘要：
深度神经网络在有监督学习任务中表现出色，但受限于对大量标注数据的需求。自监督学习作为一种有前景的替代方案应运而生，使模型能够在没有显式标签的情况下进行学习。信息理论，尤其是信息瓶颈原则，在塑造深度神经网络方面发挥了关键作用。该原则侧重于优化压缩与保留相关信息之间的权衡，为有监督环境下的高效网络设计提供了基础。然而，其在自监督学习中的确切作用和适应性仍不明确。在本研究中，我们从信息理论的视角审视了各种自监督学习方法，引入了一个统一的框架，该框架涵盖了自监督信息理论学习问题。我们将现有研究整合为一个连贯的叙述，深入探讨了当代自监督方法论，并突出了潜在的研究方向和内在挑战。此外，我们还讨论了信息理论量的经验评估及其估计方法。总体而言，本文对信息理论、自监督学习和深度神经网络的交叉领域提供了详尽的综述。
英文摘要：
Deep neural networks excel in supervised learning tasks but are constrained by the need for extensive labeled data. Self-supervised learning emerges as a promising alternative, allowing models to learn without explicit labels. Information theory, and notably the information bottleneck principle, has been pivotal in shaping deep neural networks. This principle focuses on optimizing the trade-off between compression and preserving relevant information, providing a foundation for efficient network design in supervised contexts. However, its precise role and adaptation in self-supervised learning remain unclear. In this work, we scrutinize various self-supervised learning approaches from an information-theoretic perspective, introducing a unified framework that encapsulates the \textit{self-supervised information-theoretic learning problem}. We weave together existing research into a cohesive narrative, delve into contemporary self-supervised methodologies, and spotlight potential research avenues and inherent challenges. Additionally, we discuss the empirical evaluation of information-theoretic quantities and their estimation methods. Overall, this paper furnishes an exhaustive review of the intersection of information theory, self-supervised learning, and deep neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
