We take a geometrical viewpoint and present a unifying view on supervised deep learning with the Bregman divergence loss function - this entails frequent classification and prediction tasks. Motivated by simulations we suggest that there is principally no implicit bias of vanilla stochastic gradient descent training of deep models towards "simpler" functions. Instead, we show that good generalization may be instigated by bounded spectral products over layers leading to a novel geometric regularizer. It is revealed that in deep enough models such a regularizer enables both, extreme accuracy and generalization, to be reached. We associate popular regularization techniques like weight decay, drop out, batch normalization, and early stopping with this perspective. Backed up by theory we further demonstrate that "generalization by design" is practically possible and that good generalization may be encoded into the structure of the network. We design two such easy-to-use structural regularizers that insert an additional \textit{generalization layer} into a model architecture, one with a skip connection and another one with drop-out. We verify our theoretical results in experiments on various feedforward and convolutional architectures, including ResNets, and datasets (MNIST, CIFAR10, synthetic data). We believe this work opens up new avenues of research towards better generalizing architectures.
He et al.（2015）提出的残差网络（ResNet）利用快捷连接显著降低了训练难度，从而在训练和泛化误差方面都有很大的性能提升。He等人（2015）根据经验观察到，使用快捷方式2堆叠更多层的残差块会导致更小的训练误差，而对于长度为1或3的快捷方式则不成立。我们提供了一个理论解释的唯一性的捷径2。我们证明了，无论有无非线性，通过添加深度为2的快捷方式，损失函数在零初始点处的Hessian的条件数是深度不变的，这使得训练非常深的模型并不比浅的模型更困难。更高深度的捷径最初会导致非常平坦的（高阶）稳定点，优化算法很难逃脱。然而，快捷方式1基本上等同于没有快捷方式，其具有随着层数的增加而爆炸到无穷大的条件数。我们进一步认为，由于层数趋于无穷大，它足以只看在零初始点的损失函数。广泛的实验提供伴随我们的理论结果。我们表明，使用捷径2将网络初始化为小权重比随机高斯（Xavier）初始化，正交初始化和更深深度的捷径取得了更好的结果，从最终损失，学习动态和稳定性，到Hessian的行为等各个角度来看。
Understand the Effectiveness of Shortcuts through the Lens of DCA
Yau Mathematical Sciences Center (YMSC), Tsinghua University（清华大学丘成桐数学科学中心）；Beijing Institute of Mathematical Sciences and Applications (BIMSA)（北京数学科学与应用研究院）
Towards Combating Frequency Simplicity-biased Learning for Domain Generalization
Computer Vision Institute（计算机视觉研究所）；School of Computer Science & Software Engineering（计算机科学与软件工程学院）；Shenzhen University（深圳大学）；Shenzhen Institute of Artificial Intelligence Robotics for Society（深圳人工智能与机器人社会研究院）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；University of Exeter（埃克塞特大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
MiMu: Mitigating Multiple Shortcut Learning Behavior of Transformers
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；Hefei University of Technology（合肥工业大学）；iFLYTEK AI Research (Central China), iFLYTEK Co., Ltd（iFLYTEK中央研究院，iFLYTEK公司）
Mitigating Shortcut Learning with InterpoLated Learning
Department of Computer Science and Technology, University of Cambridge（计算机科学与技术系，剑桥大学）；Department of Engineering, University of Cambridge（工程系，剑桥大学）；The Alan Turing Institute（艾伦·图灵研究所）
University of Tübingen（蒂宾根大学）；International Max Planck Research School for Intelligent Systems（国际马克斯·普朗克智能系统研究院）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
A Too-Good-to-be-True Prior to Reduce Shortcut Reliance
HSE University（俄罗斯高等经济大学）；University College London（伦敦大学学院）；Research Center Jülich（尤利希研究中心）；Heinrich Heine University Düsseldorf（海因里希·海涅大学杜塞尔多夫分校）；The Alan Turing Institute（阿兰·图灵研究所）
Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcuts
University of Michigan（密歇根大学）；Institute for Healthcare Policy & Innovation（医疗政策与创新研究所）；Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Department of Radiology（放射科）；Department of Internal Medicine（内科）
Improving deep neural network generalization and robustness to background bias via layer-wise relevance propagation optimization
Alma Mater Studiorum - University of Bologna（博洛尼亚大学）；Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia（意大利技术研究院生物分子纳米技术中心）；School of Medical Sciences, University of Campinas (UNICAMP)（坎皮纳斯大学（UNICAMP）医学院）；Istituto Italiano di Tecnologia（意大利技术研究院）
Shortcut Learning in Binary Classifier Black Boxes: Applications to Voice Anti-Spoofing and Biometrics
TCG CREST；Carnegie Mellon University（卡内基梅隆大学）；School of Computing, University of Eastern Finland（东部芬兰大学计算机学院）；University of Eastern Finland（东部芬兰大学）；University of Oulu（奥卢大学）
Layer-Aware Analysis of Catastrophic Overfitting: Revealing the Pseudo-Robust Shortcut Dependency
Sydeny AI Centre（悉尼人工智能中心）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Hong Kong Baptist University（香港浸会大学）；BNRist Center, Tsinghua University（清华大学BNRist中心）
ICON decomposition: Multivariate concept auditing reveals shortcuts that neural networks rely on
Hertie Institute for AI in Brain Health, University of Tübingen（蒂宾根大学赫蒂脑健康人工智能研究所）；Charité - Universitätsmedizin Berlin（柏林夏里特医学院）；Humboldt-Universität zu Berlin（柏林洪堡大学）；Tübingen AI Center, University of Tübingen（蒂宾根大学蒂宾根人工智能中心）；German Center for Mental Health (DZPG)（德国心理健康中心）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希赫兹研究所）；Technische Universität Berlin（柏林工业大学）
