The fruits of science are relationships made comprehensible, often by way of approximation. While deep learning is an extremely powerful way to find relationships in data, its use in science has been hindered by the difficulty of understanding the learned relationships. The Information Bottleneck (IB) is an information theoretic framework for understanding a relationship between an input and an output in terms of a trade-off between the fidelity and complexity of approximations to the relationship. Here we show that a crucial modification -- distributing bottlenecks across multiple components of the input -- opens fundamentally new avenues for interpretable deep learning in science. The Distributed Information Bottleneck throttles the downstream complexity of interactions between the components of the input, deconstructing a relationship into meaningful approximations found through deep learning without requiring custom-made datasets or neural network architectures. Applied to a complex system, the approximations illuminate aspects of the system's nature by restricting -- and monitoring -- the information about different components incorporated into the approximation. We demonstrate the Distributed IB's explanatory utility in systems drawn from applied mathematics and condensed matter physics. In the former, we deconstruct a Boolean circuit into approximations that isolate the most informative subsets of input components without requiring exhaustive search. In the latter, we localize information about future plastic rearrangement in the static structure of a sheared glass, and find the information to be more or less diffuse depending on the system's preparation. By way of a principled scheme of approximations, the Distributed IB brings much-needed interpretability to deep learning and enables unprecedented analysis of information flow through a system.
Distributed Information Bottleneck Theory for Multi-Modal Task-Aware Semantic Communication
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通信学院，华中科技大学）；Peng Cheng Laboratory（鹏城实验室）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））；School of Mechanical Engineering and Electronic Information, China University of Geosciences（机械工程与电子信息学院，中国地质大学）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）
Incorporating Retrieval-based Causal Learning with Information Bottlenecks for Interpretable Graph Neural Networks
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of Machine Intelligence and Advanced Computing, Sun Yat-sen University（中山大学机器智能与先进计算重点实验室）
Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis
Shenzhen MSU-BIT University（深圳北理莫斯科大学）；Shenzhen University（深圳大学）；University of Science and Technology Beijing（北京科技大学）；Carleton University（卡尔顿大学）；The University of British Columbia（不列颠哥伦比亚大学）
InfoBot: Transfer and Exploration via the Information Bottleneck
Mila（米拉研究所）；University of Montreal（蒙特利尔大学）；McGill University（麦吉尔大学）；Deepmind（深度思维）；Google Brain（谷歌大脑）；University College London（伦敦大学学院）；University of California, Berkeley（加州大学伯克利分校）
