Effective clinical deployment of deep learning models in healthcare demands high generalization performance to ensure accurate diagnosis and treatment planning. In recent years, significant research has focused on improving the generalization of deep learning models by regularizing the sharpness of the loss landscape. Among the optimization approaches that explicitly minimize sharpness, Sharpness-Aware Minimization (SAM) has shown potential in enhancing generalization performance on general domain image datasets. This success has led to the development of several advanced sharpness-based algorithms aimed at addressing the limitations of SAM, such as Adaptive SAM, surrogate-Gap SAM, Weighted SAM, and Curvature Regularized SAM. These sharpness-based optimizers have shown improvements in model generalization compared to conventional stochastic gradient descent optimizers and their variants on general domain image datasets, but they have not been thoroughly evaluated on medical images. This work provides a review of recent sharpness-based methods for improving the generalization of deep learning networks and evaluates the methods performance on medical breast ultrasound images. Our findings indicate that the initial SAM method successfully enhances the generalization of various deep learning models. While Adaptive SAM improves generalization of convolutional neural networks, it fails to do so for vision transformers. Other sharpness-based optimizers, however, do not demonstrate consistent results. The results reveal that, contrary to findings in the non-medical domain, SAM is the only recommended sharpness-based optimizer that consistently improves generalization in medical image analysis, and further research is necessary to refine the variants of SAM to enhance generalization performance in this field
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）
Modern deep neural networks (DNNs) have achieved state-of-the-art performances but are typically over-parameterized. The over-parameterization may result in undesirably large generalization error in the absence of other customized training strategies. Recently, a line of research under the name of Sharpness-Aware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. However, SAM-like methods incur a two-fold computational overhead of the given base optimizer (e.g. SGD) for approxim…
The goal of this work is to identify the best optimizers for deep learning in the context of cardiac image segmentation and to provide guidance on how to design segmentation networks with effective optimization strategies. Adaptive learning helps with fast convergence by starting with a larger learning rate (LR) and gradually decreasing it. Momentum optimizers are particularly effective at quickly optimizing neural networks within the accelerated schemes category. By revealing the potential interplay between these two types of algorithms (LR and momentum optimizers or momentum rate (MR) in sho…
Centre for Frontier AI Research (CFAR), A*STAR（新加坡科技研究局前沿人工智能研究中心）；National University of Singapore（新加坡国立大学）；Institute of High Performance Computing (IHPC), A*STAR（新加坡科技研究局高性能计算研究所）
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts…
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；School of Management, Xi’an Jiaotong University（西安交通大学管理学院）；School of Software, Xi’an Jiaotong University（西安交通大学软件学院）
School of Cyber Science and Technology, Sun Yat-sen University, Shenzhen Campus（网络安全科学与技术学院，中山大学深圳校区）；School of Computer Science, University of Science and Technology of China（计算机科学学院，中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）
University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；JD Explore Academy（京东探索研究院）
Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of Ad…
National University of Singapore（新加坡国立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers. However, the update rule of SAM requires two sequential (non-parallelizable) gradient computations at each step, which can double the computational overhead. In this paper, we propose a novel algorithm LookSAM - that only periodically calculates the inner gradient ascent, to significantly reduce the additional training cost of SAM. The empirical results illustrate that Look…
Network quantization is a dominant paradigm of model compression. However, the abrupt changes in quantized weights during training often lead to severe loss fluctuations and result in a sharp loss landscape, making the gradients unstable and thus degrading the performance. Recently, Sharpness-Aware Minimization (SAM) has been proposed to smooth the loss landscape and improve the generalization performance of the models. Nevertheless, directly applying SAM to the quantized models can lead to perturbation mismatch or diminishment issues, resulting in suboptimal performance. In this paper, we pro…
Xiamen University（厦门大学）；JD Explore Academy（京东探索研究院）；The University of Sydney（悉尼大学）
Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, we propose an efficient and effective training scheme coine…
Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization
School of Information Science and Technology, Beijing University of Technology（北京工业大学信息科学与技术学院）；Hangzhou Research Institute, School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院杭州研究院）；Nanchang Institute of Technology（南昌理工学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）
Fair Federated Medical Image Classification Against Quality Shift via Inter-Client Progressive State Matching
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Lassonde School of Engineering, York University（约克大学拉萨onde工程学院）
Image registration plays an important role in medical image analysis. Conventional optimization based methods provide an accurate estimation due to the iterative process at the cost of expensive computation. Deep learning methods such as learn-to-map are much faster but either iterative or coarse-to-fine approach is required to improve accuracy for handling large motions. In this work, we proposed to learn a registration optimizer via a multi-scale neural ODE model. The inference consists of iterative gradient updates similar to a conventional gradient descent optimizer but in a much faster wa…
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
Northwestern Polytechnical University（西北工业大学）；SGIT AI Lab, State Grid Corporation of China（国网SGIT人工智能实验室）；Xi’an Jiaotong University（西安交通大学）；ETH Zürich（苏黎世联邦理工学院）；Chinese University of Hong Kong（香港中文大学）
In deep learning, optimization plays a vital role. By focusing on image classification, this work investigates the pros and cons of the widely used optimizers, and proposes a new optimizer: Perturbated Unit Gradient Descent (PUGD) algorithm with extending normalized gradient operation in tensor within perturbation to update in unit space. Via a set of experiments and analyses, we show that PUGD is locally bounded updating, which means the updating from time to time is controlled. On the other hand, PUGD can push models to a flat minimum, where the error remains approximately constant, not only…
In this paper, we present UNet++, a new, more powerful architecture for medical image segmentation. Our architecture is essentially a deeply-supervised encoder-decoder network where the encoder and decoder sub-networks are connected through a series of nested, dense skip pathways. The re-designed skip pathways aim at reducing the semantic gap between the feature maps of the encoder and decoder sub-networks. We argue that the optimizer would deal with an easier learning task when the feature maps from the decoder and encoder networks are semantically similar. We have evaluated UNet++ in compari…
In this paper, we study the sharpness of a deep learning (DL) loss landscape around local minima in order to reveal systematic mechanisms underlying the generalization abilities of DL models. Our analysis is performed across varying network and optimizer hyper-parameters, and involves a rich family of different sharpness measures. We compare these measures and show that the low-pass filter-based measure exhibits the highest correlation with the generalization abilities of DL models, has high robustness to both data and label noise, and furthermore can track the double descent behavior for neur…
CLADE: Cycle Loss Augmented Degradation Enhancement for Unpaired Super-Resolution of Anisotropic Medical Images
University College London（伦敦大学学院）；Institute of Cardiovascular Science（心血管科学研究所）；Centre for Medical Imaging（医学影像中心）；Frimley Health NHS Trust（弗林利健康 NHS 信托基金）；UCL NHS Trust（伦敦大学学院 NHS 信托基金）
University of Helsinki（赫尔辛基大学）；Helsinki University Hospital（赫尔辛基大学医院）；Institute for Molecular Medicine Finland (FIMM)（芬兰分子医学研究所）
Many current neural networks for medical imaging generalise poorly to data unseen during training. Such behaviour can be caused by networks overfitting easy-to-learn, or statistically dominant, features while disregarding other potentially informative features. For example, indistinguishable differences in the sharpness of the images from two different scanners can degrade the performance of the network significantly. All neural networks intended for clinical practice need to be robust to variation in data caused by differences in imaging equipment, sample preparation and patient populations. …
Beamforming in ultrasound imaging has significant impact on the quality of the final image, controlling its resolution and contrast. Despite its low spatial resolution and contrast, delay-and-sum is still extensively used nowadays in clinical applications, due to its real-time capabilities. The most common alternatives are minimum variance method and its variants, which overcome the drawbacks of delay-and-sum, at the cost of higher computational complexity that limits its utilization in real-time applications. In this paper, we propose to perform beamforming in ultrasound imaging through a reg…
Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院，香港中文大学（深圳））；School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院和人工智能学院，香港中文大学（深圳））
How to train deep neural networks (DNNs) to generalize well is a central concern in deep learning, especially for severely overparameterized networks nowadays. In this paper, we propose an effective method to improve the model generalization by additionally penalizing the gradient norm of loss function during optimization. We demonstrate that confining the gradient norm of loss function could help lead the optimizers towards finding flat minima. We leverage the first-order approximation to efficiently implement the corresponding gradient to fit well in the gradient descent framework. In our ex…
On-the-Fly Guidance Training for Medical Image Registration
University of Leeds（利兹大学）；University of California, Irvine（加州大学欧文分校）；Tongji University（同济大学）；Huazhong University of Science and Technology（华中科技大学）；Fudan Univeristy（复旦大学）；University of California, San Diego（加州大学圣迭戈分校）
Active learning is a unique abstraction of machine learning techniques where the model/algorithm could guide users for annotation of a set of data points that would be beneficial to the model, unlike passive machine learning. The primary advantage being that active learning frameworks select data points that can accelerate the learning process of a model and can reduce the amount of data needed to achieve full accuracy as compared to a model trained on a randomly acquired data set. Multiple frameworks for active learning combined with deep learning have been proposed, and the majority of them …
A Bio-Medical Snake Optimizer System Driven by Logarithmic Surviving Global Search for Optimizing Feature Selection and its application for Disorder Recognition
Middle East University（中东大学）；Al-Balqa Applied University（巴尔卡应用大学）；Helwan University（赫勒万大学）；Guru Nanak Dev University（古鲁纳纳克德夫大学）；University of Granada（格拉纳达大学）
INRIA, CNRS, Département d’Informatique de l’Ecole Normale Supérieure / PSL, France（法国国家信息与自动化技术研究院（INRIA）、法国国家科学研究中心（CNRS）、巴黎综合理工学院计算机系 / PSL研究院）；Department of Computing, Imperial College London, United Kingdom（伦敦帝国理工学院计算机系，英国）
University of Basel（巴塞尔大学）；ETH Zürich（苏黎世联邦理工学院）；University of Oslo（奥斯陆大学）；Inria（法国国家信息与自动化研究所）；Ecole Normale Supérieure PSL Research University（巴黎高等师范学院-PSL研究大学）
本文为 SAM 及其两种变体在全批量与小批量设置下建立 SDE 连续时间模型，解释其偏好平坦极小值的机制，并证明其在现实条件下会被鞍点吸引。
Although numerous improvements have been made in the field of image segmentation using convolutional neural networks, the majority of these improvements rely on training with larger datasets, model architecture modifications, novel loss functions, and better optimizers. In this paper, we propose a new segmentation performance boosting paradigm that relies on optimally modifying the network's input instead of the network itself. In particular, we leverage the gradients of a trained segmentation network with respect to the input to transfer it to a space where the segmentation accuracy improves.…
Deep learning models have the capacity to fundamentally revolutionize medical imaging analysis, and they have particularly interesting applications in computer-aided diagnosis. We attempt to use deep learning neural networks to diagnose funduscopic images, visual representations of the interior of the eye. Recently, a few robust deep learning approaches have performed binary classification to infer the presence of a specific ocular disease, such as glaucoma or diabetic retinopathy. In an effort to broaden the applications of computer-aided ocular disease diagnosis, we propose a unifying model …
Fraunhofer Institute for Telecommunications–Heinrich Hertz Institute（弗劳恩霍夫电信研究所-海因里希·赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Experimental and Clinical Research Center（实验与临床研究中心）；Charité Medical Faculty（夏里特医学院）；Max-Delbrueck Center for Molecular Medicine（马克斯·德尔布吕克分子医学中心）；HELIOS Klinikum Berlin Buch（柏林布赫赫利俄斯医院）；Physikalisch-Technische Bundesanstalt (PTB)（德国联邦物理技术研究院（PTB））；King’s College London（伦敦国王学院）
High-resolution three-dimensional (3D) cardiovascular magnetic resonance (CMR) is a valuable medical imaging technique, but its widespread application in clinical practice is hampered by long acquisition times. Here we present a novel compressed sensing (CS) reconstruction approach using shearlets as a sparsifying transform allowing for fast 3D CMR (3DShearCS). Shearlets are mathematically optimal for a simplified model of natural images and have been proven to be more efficient than classical systems such as wavelets. Data is acquired with a 3D Radial Phase Encoding (RPE) trajectory and an it…
In this paper, we investigate whether is it possible to train a neural network directly from user inputs. We consider this approach to be highly relevant for applications in which the point of optimality is not well-defined and user-dependent. Our application is medical image denoising which is essential in fluoroscopy imaging. In this field every user, i.e. physician, has a different flavor and image quality needs to be tailored towards each individual. To address this important problem, we propose to construct a loss function derived from a forced-choice experiment. In order to make the lear…
Debunking Optimization Myths in Federated Learning for Medical Image Classification
School of Electrical Engineering, KAIST, South Korea（韩国釜山科学技术大学电气工程学院）；AI Group, AMD, United States（AMD美国人工智能小组）；Department of Applied AI, Hansung University, South Korea（韩国汉 Sung大学应用人工智能系）；Department of Computer Science, Institute of Science Tokyo, Japan（日本东京科学研究院计算机科学系）
Challenge Summary U-MedSAM: Uncertainty-aware MedSAM for Medical Image Segmentation
University at Albany, State University of New York (SUNY)（纽约州立大学奥尔巴尼分校）；College of Integrated Health Sciences and the AI Plus Institute, University at Albany, State University of New York (SUNY)（纽约州立大学奥尔巴尼分校整合健康科学学院及AI Plus研究所）；School of Physics and Electronics, Shandong Normal University（山东师范大学物理与电子科学学院）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Purdue University（普渡大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
