Multi-Instance Multi-Scale CNN for Medical Image Classification
Institute of High Performance Computing, A*STAR（A*STAR高性能计算研究所）；Artificial Intelligence Initiative, A*STAR（A*STAR人工智能倡议组织）；Singapore Eye Research Institute（新加坡眼科研究所）
Shaohua Li, Yong Liu, Xiuchao Sui, Cheng Chen, Gabriel Tjio, Daniel Shu Wei Ting, Rick Siow Mong Goh
Deep learning for medical image classification faces three major challenges: 1) the number of annotated medical images for training are usually small; 2) regions of interest (ROIs) are relatively small with unclear boundaries in the whole medical images, and may appear in arbitrary positions across the x,y (and also z in 3D images) dimensions. However often only labels of the whole images are annotated, and localized ROIs are unavailable; and 3) ROIs in medical images often appear in varying sizes (scales). We approach these three challenges with a Multi-Instance Multi-Scale (MIMS) CNN: 1) We propose a multi-scale convolutional layer, which extracts patterns of different receptive fields with a shared set of convolutional kernels, so that scale-invariant patterns are captured by this compact set of kernels. As this layer contains only a small number of parameters, training on small datasets becomes feasible; 2) We propose a "top-k pooling" to aggregate the feature maps in varying scales from multiple spatial dimensions, allowing the model to be trained using weak annotations within the multiple instance learning (MIL) framework. Our method is shown to perform well on three classification tasks involving two 3D and two 2D medical image datasets.
Convolutional neural networks have been applied to a wide variety of computer vision tasks. Recent advances in semantic segmentation have enabled their application to medical image segmentation. While most CNNs use two-dimensional kernels, recent CNN-based publications on medical image segmentation featured three-dimensional kernels, allowing full access to the three-dimensional structure of medical images. Though closely related to semantic segmentation, medical image segmentation includes specific challenges that need to be addressed, such as the scarcity of labelled data, the high class imb…
Deep neural networks have been investigated in learning latent representations of medical images, yet most of the studies limit their approach in a single supervised convolutional neural network (CNN), which usually rely heavily on a large scale annotated dataset for training. To learn image representations with less supervision involved, we propose a deep Siamese CNN (SCNN) architecture that can be trained with only binary image pair information. We evaluated the learned image representations on a task of content-based medical image retrieval using a publicly available multiclass diabetic ret…
The fast growing deep learning technologies have become the main solution of many machine learning problems for medical image analysis. Deep convolution neural networks (CNNs), as one of the most important branch of the deep learning family, have been widely investigated for various computer-aided diagnosis tasks including long-term problems and continuously emerging new problems. Image contour detection is a fundamental but challenging task that has been studied for more than four decades. Recently, we have witnessed the significantly improved performance of contour detection thanks to the de…
Systematic Integration of Attention Modules into CNNs for Accurate and Generalizable Medical Image Diagnosis
Department of Computer Science and Artificial Intelligence（计算机科学与人工智能系）；Division of Electronics and Electrical Engineering（电子与电气工程系）；Dongguk University（东国大学）
AstraZeneca（阿斯利康）；KTH Royal Institute of Technology（瑞典皇家理工学院）；Science for Life Laboratory（生命科学实验室）
Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis. Recently, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding similar levels of performance while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is time to move to transformer-based models or if we should keep working with CNNs - can we trivially switch to transformers? If so, what are the advantages and drawbacks of switching to ViTs for medical ima…
Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependenci…
École de Technologie Supérieure（高等技术学院）；Université de Nantes（南特大学）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
We propose a unified appearance model accounting for traditional shallow (i.e. 3D SIFT keypoints) and deep (i.e. CNN output layers) image feature representations, encoding respectively specific, localized neuroanatomical patterns and rich global information into a single indexing and classification framework. A novel Bayesian model combines shallow and deep features based on an assumption of conditional independence and validated by experiments indexing specific family members and general group categories in 3D MRI neuroimage data of 1010 subjects from the Human Connectome Project, including t…
Medical image analysis benefits Computer Aided Diagnosis (CADx). A fundamental analyzing approach is the classification of medical images, which serves for skin lesion diagnosis, diabetic retinopathy grading, and cancer classification on histological images. When learning these discriminative classifiers, we observe that the convolutional neural networks (CNNs) are vulnerable to distractor interference. This is due to the similar sample appearances from different categories (i.e., small inter-class distance). Existing attempts select distractors from input images by empirically estimating thei…
Cross-modality image estimation involves the generation of images of one medical imaging modality from that of another modality. Convolutional neural networks (CNNs) have been shown to be useful in identifying, characterising and extracting image patterns. Generative adversarial networks (GANs) use CNNs as generators and estimated images are discriminated as true or false based on an additional network. CNNs and GANs within the image estimation framework may be considered more generally as deep learning approaches, since imaging data tends to be large, leading to a larger number of network wei…
University College London（伦敦大学学院）；University Hospitals KU Leuven（鲁汶大学附属医院）
Accurate medical image segmentation is essential for diagnosis, surgical planning and many other applications. Convolutional Neural Networks (CNNs) have become the state-of-the-art automatic segmentation methods. However, fully automatic results may still need to be refined to become accurate and robust enough for clinical use. We propose a deep learning-based interactive segmentation method to improve the results obtained by an automatic CNN and to reduce user interactions during refinement for higher accuracy. We use one CNN to obtain an initial automatic segmentation, on which user interact…
This paper presents a novel unsupervised segmentation method for 3D medical images. Convolutional neural networks (CNNs) have brought significant advances in image segmentation. However, most of the recent methods rely on supervised learning, which requires large amounts of manually annotated data. Thus, it is challenging for these methods to cope with the growing amount of medical images. This paper proposes a unified approach to unsupervised deep representation learning and clustering for segmentation. Our proposed method consists of two phases. In the first phase, we learn deep feature repr…
University of Copenhagen（哥本哈根大学）；Eindhoven University of Technology（埃因霍温理工大学）；Hvidovre Hospital（维兹奥勒医院）；Bispebjerg Hospital（比斯佩比约医院）；Erasmus MC - University Medical Center Rotterdam（伊拉斯谟医学中心——鹿特丹大学医学中心）
Supervised feature learning using convolutional neural networks (CNNs) can provide concise and disease relevant representations of medical images. However, training CNNs requires annotated image data. Annotating medical images can be a time-consuming task and even expert annotations are subject to substantial inter- and intra-rater variability. Assessing visual similarity of images instead of indicating specific pathologies or estimating disease severity could allow non-experts to participate, help uncover new patterns, and possibly reduce rater variability. We consider the task of assessing e…
University Medical Center Utrecht（乌得勒支大学医学中心）；Eindhoven University of Technology（埃因霍温理工大学）
Automatic segmentation of medical images is an important task for many clinical applications. In practice, a wide range of anatomical structures are visualised using different imaging modalities. In this paper, we investigate whether a single convolutional neural network (CNN) can be trained to perform different segmentation tasks. A single CNN is trained to segment six tissues in MR brain images, the pectoral muscle in MR breast images, and the coronary arteries in cardiac CTA. The CNN therefore learns to identify the imaging modality, the visualised anatomical structures, and the tissue clas…
Expert-Like Reparameterization of Heterogeneous Pyramid Receptive Fields in Efficient CNNs for Fair Medical Image Classification
Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology（可信自主系统研究所和南方科技大学计算机科学与工程系）；Center for High Performance Computing and Shenzhen Key Laboratory of Intelligent Bioinformatics, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（高性能计算中心和深圳智能生物信息学重点实验室，深圳先进技术研究院，中国科学院）；Tomey Corporation（Tomey公司）；Changchun University（长春大学）；School of computer science, University of Nottingham Ningbo China（Nottingham Ningbo中国大学计算机学院）；School of Ophthalmology and Optometry, Wenzhou Medical University（眼科学与视光学学院，温州医科大学）
Eindhoven University of Technology（埃因霍温理工大学）；Erasmus Medical Center（伊拉斯谟医学中心）；University of Copenhagen（哥本哈根大学）；University Medical Center Utrecht（乌得勒支大学医学中心）
Machine learning (ML) algorithms have made a tremendous impact in the field of medical imaging. While medical imaging datasets have been growing in size, a challenge for supervised ML algorithms that is frequently mentioned is the lack of annotated data. As a result, various methods which can learn with less/other types of supervision, have been proposed. We review semi-supervised, multiple instance, and transfer learning in medical imaging, both in diagnosis/detection or segmentation tasks. We also discuss connections between these learning scenarios, and opportunities for future research.
The George Washington University Medical Center（乔治华盛顿大学医学中心）
Recently, deep learning has become much more popular in computer vision area. The Convolution Neural Network (CNN) has brought a breakthrough in images segmentation areas, especially, for medical images. In this regard, U-Net is the predominant approach to medical image segmentation task. The U-Net not only performs well in segmenting multimodal medical images generally, but also in some tough cases of them. However, we found that the classical U-Net architecture has limitation in several aspects. Therefore, we applied modifications: 1) designed efficient CNN architecture to replace encoder an…
Imaging Biomarkers and Computer-Aided Diagnosis Laboratory（影像生物标志物与计算机辅助诊断实验室）；Center for Infectious Disease Imaging（传染病成像中心）；Clinical Image Processing Service, Radiology and Imaging Sciences Department, National Institutes of Health Clinical Center（美国国立卫生研究院临床中心放射与影像科学系临床图像处理服务部）
Remarkable progress has been made in image recognition, primarily due to the availability of large-scale annotated datasets and the revival of deep CNN. CNNs enable learning data-driven, highly representative, layered hierarchical image features from sufficient training data. However, obtaining datasets as comprehensively annotated as ImageNet in the medical imaging domain remains a challenge. There are currently three major techniques that successfully employ CNNs to medical image classification: training the CNN from scratch, using off-the-shelf pre-trained CNN features, and conducting unsup…
The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) …
Institute of Mechatronic Systems, Leibniz Universität Hannover（汉诺威莱布尼茨大学机电系统研究所）
We present deformable unsupervised medical image registration using a randomly-initialized deep convolutional neural network (CNN) as regularization prior. Conventional registration methods predict a transformation by minimizing dissimilarities between an image pair. The minimization is usually regularized with manually engineered priors, which limits the potential of the registration. By learning transformation priors from a large dataset, CNNs have achieved great success in deformable registration. However, learned methods are restricted to domain-specific data and the required amounts of me…
CFFormer: Cross CNN-Transformer Channel Attention and Spatial Feature Fusion for Improved Segmentation of Heterogeneous Medical Images
School of Computer Science, University of Nottingham Ningbo China（Nottingham 中国 Nottingham 理工学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究所，中国科学院）；School of Mathematical and Computational Sciences, Massey University（数学与计算科学学院，马斯sey 大学）；University of Nottingham, United Kingdom（Nottingham 大学，英国）
Chinese Academy of Sciences（中国科学院）；Johns Hopkins Medical Institutes（约翰斯·霍普金斯医学院）；Vanderbilt University（范德堡大学）；University of Science and Technology of China（中国科学技术大学）
Transformer, the latest technological advance of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that cha…
Macquarie University（麦考瑞大学）；University of Technology Sydney（悉尼科技大学）；The University of Sydney（悉尼大学）
Automatic tumor segmentation is a crucial step in medical image analysis for computer-aided diagnosis. Although the existing methods based on convolutional neural networks (CNNs) have achieved the state-of-the-art performance, many challenges still remain in medical tumor segmentation. This is because regular CNNs can only exploit translation invariance, ignoring further inherent symmetries existing in medical images such as rotations and reflections. To mitigate this shortcoming, we propose a novel group equivariant segmentation framework by encoding those inherent symmetries for learning mor…
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: \emph{Can the use of pre-trained deep CNNs with s…
Tel Aviv University（特拉维夫大学）；The Chaim Sheba Medical Center（夏姆谢巴医疗中心）；Bar-Ilan University（巴伊兰大学）
Deep learning methods, and in particular convolutional neural networks (CNNs), have led to an enormous breakthrough in a wide range of computer vision tasks, primarily by using large-scale annotated datasets. However, obtaining such datasets in the medical domain remains a challenge. In this paper, we present methods for generating synthetic medical images using recently presented deep learning Generative Adversarial Networks (GANs). Furthermore, we show that generated medical images can be used for synthetic data augmentation, and improve the performance of CNN for medical image classificatio…
Iran University of Science and Technology（伊朗科技大学）；West Virginia University（西弗吉尼亚大学）
Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism …
University of Sydney（悉尼大学）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）；Shanghai Jiao Tong University（上海交通大学）
Medical image analysis using supervised deep learning methods remains problematic because of the reliance of deep learning methods on large amounts of labelled training data. Although medical imaging data repositories continue to expand there has not been a commensurate increase in the amount of annotated data. Hence, we propose a new unsupervised feature learning method that learns feature representations to then differentiate dissimilar medical images using an ensemble of different convolutional neural networks (CNNs) and K-means clustering. It jointly learns feature representations and clus…
Probabilistic smooth attention for deep multiple instance learning in medical imaging
Department of Computer Science and AI, University of Granada（计算机科学与人工智能系，格拉纳达大学）；Department of Statistics and Operations Research, University of Granada（统计与运筹学系，格拉纳达大学）；Research Centre for Information and Communication Technologies (CITIC), University of Granada（信息与通信技术研究中心（CITIC），格拉纳达大学）；Department of Electrical and Computer Engineering, Northwestern University（电气与计算机工程系，西北大学）；Center for Computational Imaging and Signal Analytics in Medicine, Northwestern University（医学计算成像与信号分析中心，西北大学）
University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；UNSW（新南威尔士大学）
Gigapixel medical images provide massive data, both morphological textures and spatial information, to be mined. Due to the large data scale in histology, deep learning methods play an increasingly significant role as feature extractors. Existing solutions heavily rely on convolutional neural networks (CNNs) for global pixel-level analysis, leaving the underlying local geometric structure such as the interaction between cells in the tumor microenvironment unexplored. The topological structure in medical images, as proven to be closely related to tumor evolution, can be well characterized by gr…
The Hong Kong University of Science and Technology（香港科技大学）
Affine registration is indispensable in a comprehensive medical image registration pipeline. However, only a few studies focus on fast and robust affine registration algorithms. Most of these studies utilize convolutional neural networks (CNNs) to learn joint affine and non-parametric registration, while the standalone performance of the affine subnetwork is less explored. Moreover, existing CNN-based affine registration approaches focus either on the local misalignment or the global orientation and position of the input to predict the affine transformation matrix, which are sensitive to spati…
Multiple instance (MI) learning with a convolutional neural network enables end-to-end training in the presence of weak image-level labels. We propose a new method for aggregating predictions from smaller regions of the image into an image-level classification by using the quantile function. The quantile function provides a more complete description of the heterogeneity within each image, improving image-level classification. We also adapt image augmentation to the MI framework by randomly selecting cropped regions on which to apply MI aggregation during each epoch of training. This provides a…
Regularization in convolutional neural networks (CNNs) is usually addressed with dropout layers. However, dropout is sometimes detrimental in the convolutional part of a CNN as it simply sets to zero a percentage of pixels in the feature maps, adding unrepresentative examples during training. Here, we propose a CNN layer that performs regularization by applying random rotations of reflections to a small percentage of feature maps after every convolutional layer. We prove how this concept is beneficial for images with orientational symmetries, such as in medical images, as it provides a certain…
Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging
Junia（朱尼亚学院）；Centrale Lille（里尔中央理工学院）；University of the Basque Country UPV/EHU（巴斯克大学 UPV/EHU）；Université Polytechnique Hauts-de-France（上法兰西理工大学）；Université de Lille（里尔大学）；Ho Chi Minh City Open University（胡志明市开放大学）
Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
Fudan University, China（复旦大学）；University of Queensland, Australia（昆士兰大学）；Shanghai Academy of AI for Science, China（上海人工智能科学研究院）；Huashan Hospital, National Center for Neurological Disorders, Fudan University, China（华山医院，国家神经系统疾病中心，复旦大学）；Bioinformatics Institute (BII), Agency for Science, Technology and Research (A*STAR), Singapore（生物信息研究所（BII），科技研究局（A*STAR），新加坡）
Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis
Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（里斯本高等理工学院）；Institute of Computing（计算研究所）；University of Campinas（坎皮纳斯大学）；Lisbon ELLIS Unit（里斯本ELLIS单元）
Deformable registration continues to be one of the key challenges in medical image analysis. While iconic registration methods have started to benefit from the recent advances in medical deep learning, the same does not yet apply for the registration of point sets, e.g. registration based on surfaces, keypoints or landmarks. This is mainly due to the restriction of the convolution operator in modern CNNs to densely gridded input. However, with the newly developed methods from the field of geometric deep learning suitable tools are now emerging, which enable powerful analysis of medical data on…
DExTeR: Weakly Semi-Supervised Object Detection with Class and Instance Experts for Medical Imaging
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、法国）；University Hospital of Strasbourg, France（斯特拉斯堡大学医院、法国）
University of Toronto（多伦多大学）；Lunenfeld-Tanenbaum Research Institute, Sinai Health System（西奈卫生系统卢嫩菲尔德-塔南鲍姆研究所）；McMaster University（麦克马斯特大学）；Hamilton Health Sciences（汉密尔顿健康科学中心）；Sunnybrook Health Sciences Centre（桑尼布鲁克健康科学中心）；Sunnybrook Research Institute（桑尼布鲁克研究所）
Cox proportional hazard model (CPH) is commonly used in clinical research for survival analysis. In quantitative medical imaging (radiomics) studies, CPH plays an important role in feature reduction and modeling. However, the underlying linear assumption of CPH model limits the prognostic performance. In addition, the multicollinearity of radiomic features and multiple testing problem further impedes the CPH models performance. In this work, using transfer learning, a convolutional neural network (CNN) based survival model was built and tested on preoperative CT images of resectable Pancreatic…
From CNN to Transformer: A Review of Medical Image Segmentation Models
University of Electronic Science and Technology of China（电子科技大学）；Deyang People’s Hospital（德阳市人民医院）；Sichuan Provincial People’s Hospital（四川省人民医院）；Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital（中国科学院四川转化医学研究医院）
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
Recently, self-supervised instance discrimination methods have achieved significant success in learning visual representations from unlabeled photographic images. However, given the marked differences between photographic and medical images, the efficacy of instance-based objectives, focusing on learning the most discriminative global features in the image (i.e., wheels in bicycle), remains unknown in medical imaging. Our preliminary analysis showed that high global similarity of medical images in terms of anatomy hampers instance discrimination methods for capturing a set of distinct features…
Does Bigger Mean Better? Comparitive Analysis of CNNs and Biomedical Vision Language Modles in Medical Diagnosis
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Independent Researcher（独立研究者）；Duke University（杜克大学）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；Georgia Institute of Technology（佐治亚理工学院）；University of California, Irvine（加州大学 Irvine 分校）
Transformer, a model comprising attention-based encoder-decoder architecture, have gained prevalence in the field of natural language processing (NLP) and recently influenced the computer vision (CV) space. The similarities between computer vision and medical imaging, reviewed the question among researchers if the impact of transformers on computer vision be translated to medical imaging? In this paper, we attempt to provide a comprehensive and recent review on the application of transformers in medical imaging by; describing the transformer model comparing it with a diversity of convolutional…
Public Health Ontario（安大略省公共卫生局）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Deep learning and knowledge transfer techniques have permeated the field of medical imaging and are considered as key approaches for revolutionizing diagnostic imaging practices. However, there are still challenges for the successful integration of deep learning into medical imaging tasks due to a lack of large annotated imaging data. To address this issue, we propose a teacher-student learning framework to transfer knowledge from a carefully pre-trained convolutional neural network (CNN) teacher to a student CNN. In this study, we explore the performance of knowledge transfer in the medical i…
Optimization of Convolutional Neural Network Hyperparameter for Medical Image Diagnosis using Metaheuristic Algorithms: A short Recent Review (2019-2022)
Universiti Sains Malaysia（马来西亚理科大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；University of Information Technology and Communications（信息技术与通信大学）
Variable-frame CNNLSTM for Breast Nodule Classification using Ultrasound Videos
Beijing Normal University（北京师范大学）；Xi’an Jiaotong University（西安交通大学）；The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）；Xiangya Hospital, Central South University（中南大学湘雅医院）；Frontline Intelligent Technology (Nanjing) Co., Ltd.（南京前沿智能科技有限公司）；The Second Hospital of Hebei Medical University（河北医科大学第二医院）
The Hong Kong Polytechnic University（香港理工大学）；The University of Tokyo（东京大学）
Spatial attention has been introduced to convolutional neural networks (CNNs) for improving both their performance and interpretability in visual tasks including image classification. The essence of the spatial attention is to learn a weight map which represents the relative importance of activations within the same layer or channel. All existing attention mechanisms are local attentions in the sense that weight maps are image-specific. However, in the medical field, there are cases that all the images should share the same weight map because the set of images record the same kind of symptom r…
