Residual connections are one of the most important components in neural network architectures for mitigating the vanishing gradient problem and facilitating the training of much deeper networks. One possible explanation for how residual connections aid deeper network training is by promoting feature reuse. However, we identify and analyze the limitations of feature reuse with vanilla residual connections. To address these limitations, we propose modifications in training methods. Specifically, we provide an additional opportunity for the model to learn feature reuse with residual connections through two types of iterations during training. The first type of iteration involves using droppath, which enforces feature reuse by randomly dropping a subset of layers. The second type of iteration focuses on training the dropped parts of the model while freezing the undropped parts. As a result, the dropped parts learn in a way that encourages feature reuse, as the model relies on the undropped parts with feature reuse in mind. Overall, we demonstrated performance improvements in models with residual connections for image classification in certain cases.
StablePDENet: Enhancing Neural Operator Stability through Physics-Informed Residual-Sensitivity Regularization
Department of Mathematics, The Hong Kong University of Science and Technology(香港科技大学数学系);Algorithms of Machine Learning and Autonomous Driving Research Lab(机器学习与自动驾驶算法研究实验室);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(香港科技大学深圳-香港协同创新研究院)
Morphological Operation Residual Blocks: Enhancing 3D Morphological Feature Representation in Convolutional Neural Networks for Semantic Segmentation of Medical Images
Enhancing Time-Series Anomaly Detection by Integrating Spectral-Residual Bottom-Up Attention with Reservoir Computing
Graduate School of Information and Computer Science(信息与计算机科学研究生学校);Chiba Institute of Technology(千叶技术大学);Department of Computer Science(计算机科学系);National Center of Neurology and Psychiatry(国立神经精神医学研究中心);Research Center for Mathematical Engineering(数学工程研究中心);International Research Center for Neurointelligence(神经智能国际研究中心);Institute for AI and Beyond(人工智能与超越研究所)
Enhancing mTBI Diagnosis with Residual Triplet Convolutional Neural Network Using 3D CT
School of Electrical Engineering and Computer Science, University of Queensland(昆士兰大学电气工程与计算机科学学院);The Centre for Advanced Imaging, Australian Institute for Bioengineering and Nanotechnology, University of Queensland(昆士兰大学高级成像中心,澳大利亚生物工程与纳米技术研究所);ARC Training Centre for Innovation in Biomedical Imaging Technology(ARC生物医学成像技术创新培训中心)
RES-SE-NET: Boosting Performance of Resnets by Enhancing Bridge-connections
Sri Sathya Sai Institute of Higher Learning(斯里萨蒂亚赛高等教育学院);Department of Mathematics and Computer Science(数学与计算机科学系);Prasanthi Nilayam, AP, India(普拉萨蒂尼莱亚姆,安得拉邦,印度)
深度卷积神经网络(CNN)在各种各样的应用中取得了惊人的成果。然而,由于有限的内存和计算资源,在移动或嵌入式设备上使用这些模型是困难的。最近,反转残差块成为紧凑CNN架构设计的主要解决方案。在这项工作中,我们全面研究了现有的设计概念,重新思考的功能特性的两个逐点卷积的倒残差。我们提出了一种新颖的设计,称为不对称瓶颈。准确地说,我们调整了第一个逐点卷积维度,通过特征重用来丰富信息流,并将节省的计算迁移到第二个逐点卷积。通过这样做,我们可以在不增加计算开销的情况下进一步提高精度。非对称瓶颈可以被采用作为现有CNN块的直接替代。因此,我们可以根据适当的深度和宽度条件轻松地堆叠这些块来创建AsymmNet。大量的实验表明,我们提出的块设计比移动网络的原始反向剩余瓶颈更有益,特别是对于那些在<220 M MAds范围内的超轻型CNN。代码可在https://github.com/Spark001/AsymmNet上获得