In this article, we perform a review of the state-of-the-art of hybrid machine learning in medical imaging. We start with a short summary of the general developments of the past in machine learning and how general and specialized approaches have been in competition in the past decades. A particular focus will be the theoretical and experimental evidence pro and contra hybrid modelling. Next, we inspect several new developments regarding hybrid machine learning with a particular focus on so-called known operator learning and how hybrid approaches gain more and more momentum across essentially all applications in medical imaging and medical image analysis. As we will point out by numerous examples, hybrid models are taking over in image reconstruction and analysis. Even domains such as physical simulation and scanner and acquisition design are being addressed using machine learning grey box modelling approaches. Towards the end of the article, we will investigate a few future directions and point out relevant areas in which hybrid modelling, meta learning, and other domains will likely be able to drive the state-of-the-art ahead.
Duke University（杜克大学）；Carnegie Mellon University（卡内基梅隆大学）；Petuum, Inc.（Petuum公司）
Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an …
A Distributed Hybrid Quantum Convolutional Neural Network for Medical Image Classification
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；Joint International Research Laboratory of Intelligent Perception and Computation（智能感知与计算联合国际研究实验室）；Collaborative Innovation Center of Quantum Information of Shaanxi Province（陕西省量子信息协同创新中心）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）
The main focus of image mining in the proposed method is concerned with the classification of brain tumor in the CT scan brain images. The major steps involved in the system are: pre-processing, feature extraction, association rule mining and hybrid classifier. The pre-processing step has been done using the median filtering process and edge features have been extracted using canny edge detection technique. The two image mining approaches with a hybrid manner have been proposed in this paper. The frequent patterns from the CT scan images are generated by frequent pattern tree (FP-Tree) algorit…
the University of Hong Kong（香港大学）；China Agricultural University（中国农业大学）；Deepwise Healthcare（深睿医疗）
During clinical practice, radiologists often use attributes, e.g. morphological and appearance characteristics of a lesion, to aid disease diagnosis. Effectively modeling attributes as well as all relationships involving attributes could boost the generalization ability and verifiability of medical image diagnosis algorithms. In this paper, we introduce a hybrid neuro-probabilistic reasoning algorithm for verifiable attribute-based medical image diagnosis. There are two parallel branches in our hybrid algorithm, a Bayesian network branch performing probabilistic causal relationship reasoning a…
Medical imaging is the technique used to create images of the human body or parts of it for clinical purposes. Medical images always have large sizes and they are commonly corrupted by single or multiple noise type at the same time, due to various reasons, these two reasons are the triggers for moving toward parallel image processing to find alternatives of image de-noising techniques. This paper presents a parallel hybrid filter implementation for gray scale medical image de-noising. The hybridization is between adaptive median and wiener filters. Parallelization is implemented on the adaptiv…
Capsule-ConvKAN: A Hybrid Neural Approach to Medical Image Classification
Department of Cybernetics and Artificial Intelligence, Technical University of Košice（控制论与人工智能系，科希斯技术大学）；Institute of Information Science, University of Miskolc（信息科学研究所，米什科尔特大学）；Department of Computer Science and Engineering, University of Connecticut（计算机科学与工程系，康涅狄格大学）
HybridMamba: A Dual-domain Mamba for 3D Medical Image Segmentation
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Department of Radiology, Fudan University Shanghai Cancer Center（复旦大学上海肿瘤中心放射科）；Department of Oncology, Shanghai Medical College, Fudan University（复旦大学上海医学院肿瘤科）；The Hong Kong University of Science and Technology（香港科学与技术大学）
HySparK: Hybrid Sparse Masking for Large Scale Medical Image Pre-Training
Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）；Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）
Vision Models for Medical Imaging: A Hybrid Approach for PCOS Detection from Ultrasound Scans
Department of CSE, CCN University of Science & Technology（计算机科学与工程系，CCN科学与技术大学）；Department of EEE,International Islamic University Chittagong（电子工程系，国际伊斯兰大学恰tagong分校）；Faculty of Engineering, Multimedia University（工程学院，多媒体大学）；Department of CSE, Stamford University of Bangladesh（计算机科学与工程系，斯塔福德大学孟加拉国分校）；Department of Biology, Lucknow University（生物学系，拉胡尔大学）；Department of CSE, Bangladesh Army International University of Science & Technology（计算机科学与工程系，孟加拉国军队国际科学与技术大学）
Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging
Junia（朱尼亚学院）；Centrale Lille（里尔中央理工学院）；University of the Basque Country UPV/EHU（巴斯克大学 UPV/EHU）；Université Polytechnique Hauts-de-France（上法兰西理工大学）；Université de Lille（里尔大学）；Ho Chi Minh City Open University（胡志明市开放大学）
Advancing Medical Radiograph Representation Learning: A Hybrid Pre-training Paradigm with Multilevel Semantic Granularity
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The University of Georgia（佐治亚大学）；Northwestern Polytechnical University（西北工业大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Institute for Signals, Systems and Computational Intelligence, sinc(i) CONICET-UNL（信号、系统与计算智能研究所，sinc(i) 阿根廷国家科学技术研究委员会-国立利托拉尔大学）；Hospital Italiano de Buenos Aires（布宜诺斯艾利斯意大利医院）；Universidad Tecnológica Nacional（国立技术大学）
Anatomical segmentation is a fundamental task in medical image computing, generally tackled with fully convolutional neural networks which produce dense segmentation masks. These models are often trained with loss functions such as cross-entropy or Dice, which assume pixels to be independent of each other, thus ignoring topological errors and anatomical inconsistencies. We address this limitation by moving from pixel-level to graph representations, which allow to naturally incorporate anatomical constraints by construction. To this end, we introduce HybridGNet, an encoder-decoder neural archit…
Recent works on Multimodal 3D Computer-aided diagnosis have demonstrated that obtaining a competitive automatic diagnosis model when a 3D convolution neural network (CNN) brings more parameters and medical images are scarce remains nontrivial and challenging. Considering both consistencies of regions of interest in multimodal images and diagnostic accuracy, we propose a novel mutual attention-based hybrid dimensional network for MultiModal 3D medical image classification (MMNet). The hybrid dimensional network integrates 2D CNN with 3D convolution modules to generate deeper and more informativ…
Latents of latents to delineate pixels: hybrid Matryoshka autoencoder-to-U-Net pairing for segmenting large medical images in GPU-poor and low-data regimes
School of Mathematics and Computer Science, Institute of Business Administration（商业管理学院数学与计算机科学学院）
D-TrAttUnet: Toward Hybrid CNN-Transformer Architecture for Generic and Subtle Segmentation in Medical Images
Junia（朱尼亚学院）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；Université Polytechnique Hauts-de-France（上法兰西理工大学）；National Research Council of Italy（意大利国家研究委员会）；University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））；IKERBASQUE（伊克尔巴斯克基金会）；Université de Lille（里尔大学）
Multimodal Human-AI Synergy for Medical Imaging Quality Control: A Hybrid Intelligence Framework with Adaptive Dataset Curation and Closed-Loop Evaluation
Zhongda Hospital（中大医院）；School of Medicine, Southeast University（东南大学医学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Diffusion Low Rank Hybrid Reconstruction for Sparse View Medical Imaging
Shenzhen MSU-BIT University（深圳MSU-BIT大学）；China Media Group（中国媒体集团）；School of Mathematics and Statistics, Beijing Institute of Technology（北京理工大学数学与统计学院）
Hybrid Deep Learning Framework for Classification of Kidney CT Images: Diagnosis of Stones, Cysts, and Tumors
School of Engineering & Technology, BML Munjal University（BML Munjal大学工程与技术学院）；Center for Advanced Data and Computational Science, BML Munjal University（BML Munjal大学先进数据与计算科学中心）
Automatic image segmentation becomes very crucial for tumor detection in medical image processing.In general, manual and semi automatic segmentation techniques require more time and knowledge. However these drawbacks had overcome by automatic segmentation still there needs to develop more appropriate techniques for medical image segmentation. Therefore, we proposed hybrid approach based image segmentation using the combined features of region growing and threshold based segmentation techniques. It is followed by pre-processing stage to provide an accurate brain tumor extraction by the help of …
Iran University of Science and Technology（伊朗科技大学）；West Virginia University（西弗吉尼亚大学）
Convolutional Neural Networks (CNNs) have advanced existing medical systems for automatic disease diagnosis. However, there are still concerns about the reliability of deep medical diagnosis systems against the potential threats of adversarial attacks since inaccurate diagnosis could lead to disastrous consequences in the safety realm. In this study, we propose a highly robust yet efficient CNN-Transformer hybrid model which is equipped with the locality of CNNs as well as the global connectivity of vision Transformers. To mitigate the high quadratic complexity of the self-attention mechanism …
A Hybrid Fully Convolutional CNN-Transformer Model for Inherently Interpretable Disease Detection from Retinal Fundus Images
Hertie Institute for AI in Brain Health, University of Tübingen, Germany（人工智能与脑健康赫尔特研究所，图宾根大学，德国）；Tübingen AI Center, University of Tübingen, Germany（图宾根人工智能中心，图宾根大学，德国）
COMSATS University Islamabad（COMSATS伊斯兰堡大学）；Virtual University of Pakistan（巴基斯坦虚拟大学）；Cardiff University（卡迪夫大学）
The deep learning-based analysis of medical images suffers from data scarcity because of high annotation costs and privacy concerns. Researchers in this domain have used transfer learning to avoid overfitting when using complex architectures. However, the domain differences between pre-training and downstream data hamper the performance of the downstream task. Some recent studies have successfully used domain-adaptive pre-training (DAPT) to address this issue. In DAPT, models are initialized with the generic dataset pre-trained weights, and further pre-training is performed using a moderately …
Is the medical image segmentation problem solved? A survey of current developments and future directions
University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；University of Pennsylvania（宾夕法尼亚大学）；Cleveland Clinic（克利夫兰诊所）；Mayo Clinic（梅奥诊所）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）
Image denoising is an important pre-processing step in medical image analysis. Different algorithms have been proposed in past three decades with varying denoising performances. More recently, having outperformed all conventional methods, deep learning based models have shown a great promise. These methods are however limited for requirement of large training sample size and high computational costs. In this paper we show that using small sample size, denoising autoencoders constructed using convolutional layers can be used for efficient denoising of medical images. Heterogeneous images can be…
Despite the surge of deep learning in the past decade, some users are skeptical to deploy these models in practice due to their black-box nature. Specifically, in the medical space where there are severe potential repercussions, we need to develop methods to gain confidence in the models' decisions. To this end, we propose a novel medical imaging generative adversarial framework, medXGAN (medical eXplanation GAN), to visually explain what a medical classifier focuses on in its binary predictions. By encoding domain knowledge of medical images, we are able to disentangle anatomical structure an…
In the past ten years, with the help of deep learning, especially the rapid development of deep neural networks, medical image analysis has made remarkable progress. However, how to effectively use the relational information between various tissues or organs in medical images is still a very challenging problem, and it has not been fully studied. In this thesis, we propose two novel solutions to this problem based on deep relational learning. First, we propose a context-aware fully convolutional network that effectively models implicit relation information between features to perform medical i…
Decision support tools that rely on supervised learning require large amounts of expert annotations. Using past radiological reports obtained from hospital archiving systems has many advantages as training data above manual single-class labels: they are expert annotations available in large quantities, covering a population-representative variety of pathologies, and they provide additional context to pathology diagnoses, such as anatomical location and severity. Learning to auto-generate such reports from images present many challenges such as the difficulty in representing and generating long…
Imaging has occupied a huge role in the management of patients, whether hospitalized or not. Depending on the patients clinical problem, a variety of imaging modalities were available for use. This gave birth of the annotation of medical image process. The annotation is intended to image analysis and solve the problem of semantic gap. The reason for image annotation is due to increase in acquisition of images. Physicians and radiologists feel better while using annotation techniques for faster remedy in surgery and medicine due to the following reasons: giving details to the patients, searchin…
Systematic Integration of Attention Modules into CNNs for Accurate and Generalizable Medical Image Diagnosis
Department of Computer Science and Artificial Intelligence（计算机科学与人工智能系）；Division of Electronics and Electrical Engineering（电子与电气工程系）；Dongguk University（东国大学）
Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it can not deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependenci…
University of Oklahoma（俄克拉荷马大学）；ShanghaiTech University（上海科技大学）；University of Oklahoma Health Sciences Center（俄克拉荷马大学健康科学中心）
Deep learning has received extensive research interest in developing new medical image processing algorithms, and deep learning based models have been remarkably successful in a variety of medical imaging tasks to support disease detection and diagnosis. Despite the success, the further improvement of deep learning models in medical image analysis is majorly bottlenecked by the lack of large-sized and well-annotated datasets. In the past five years, many studies have focused on addressing this challenge. In this paper, we reviewed and summarized these recent studies to provide a comprehensive …
In the past few years, deep learning-based methods have demonstrated enormous success for solving inverse problems in medical imaging. In this work, we address the following question:\textit{Given a set of measurements obtained from real imaging experiments, what is the best way to use a learnable model and the physics of the modality to solve the inverse problem and reconstruct the latent image?} Standard supervised learning based methods approach this problem by collecting data sets of known latent images and their corresponding measurements. However, these methods are often impractical due …
Due to the lack of expertise for medical image annotation, the investigation of label-efficient methodology for medical image segmentation becomes a heated topic. Recent progresses focus on the efficient utilization of weak annotations together with few strongly-annotated labels so as to achieve comparable segmentation performance in many unprofessional scenarios. However, these approaches only concentrate on the supervision inconsistency between strongly- and weakly-annotated instances but ignore the instance inconsistency inside the weakly-annotated instances, which inevitably leads to perfo…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）
Deep learning has made a remarkable impact in the field of natural image processing over the past decade. Consequently, there is a great deal of interest in replicating this success across unsolved tasks in related domains, such as medical image analysis. Core to medical image analysis is the task of semantic segmentation which enables various clinical workflows. Due to the challenges inherent in manual segmentation, many decades of research have been devoted to discovering extensible, automated, expert-level segmentation techniques. Given the groundbreaking performance demonstrated by recent …
Viterbi School of Engineering, University of Southern California（南加州大学维特比工程学院）；The Hong Kong Polytechnic University（香港理工大学）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Beijing Jiaotong University（北京交通大学）；The Broad institute of MIT and Harvard（麻省理工学院与哈佛大学博德研究所）；Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
Sometimes electrical medical records are restricted and difficult to centralize for machine learning, which could only be trained in distributed manner that involved many institutions in the process. However, sometimes some institutions are likely to figure out the private data used for training certain models based on the parameters they obtained, which is a violation of privacy and certain regulations. Under those circumstances, we develop an algorithm, called 'federated machine learning with anonymous random hybridization'(abbreviated as 'FeARH'), using mainly hybridization algorithm to eli…
CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61部门）；Queensland University of Technology（昆士兰科技大学）
With the advances of data-driven machine learning research, a wide variety of prediction problems have been tackled. It has become critical to explore how machine learning and specifically deep learning methods can be exploited to analyse healthcare data. A major limitation of existing methods has been the focus on grid-like data; however, the structure of physiological recordings are often irregular and unordered which makes it difficult to conceptualise them as a matrix. As such, graph neural networks have attracted significant attention by exploiting implicit information that resides in a b…
Kharkiv National University of Radio Electronics（哈尔科夫国立无线电电子大学）；Kharkiv Medical Academia of Post-Graduate Education（哈尔科夫医学研究生教育学院）；Uzhhorod National University（乌日霍罗德国立大学）
The hybrid clustering-classification neural network is proposed. This network allows increasing a quality of information processing under the condition of overlapping classes due to the rational choice of a learning rate parameter and introducing a special procedure of fuzzy reasoning in the clustering process, which occurs both with an external learning signal (supervised) and without the one (unsupervised). As similarity measure neighborhood function or membership one, cosine structures are used, which allow to provide a high flexibility due to self-learning-learning process and to provide s…
International Institute of Information Technology（国际信息技术研究所）
Mining relationships between treatment(s) and medical problem(s) is vital in the biomedical domain. This helps in various applications, such as decision support system, safety surveillance, and new treatment discovery. We propose a deep learning approach that utilizes both word level and sentence-level representations to extract the relationships between treatment and problem. While deep learning techniques demand a large amount of data for training, we make use of a rule-based system particularly for relationship classes with fewer samples. Our final relations are derived by jointly combining…
MaxViT-UNet: Multi-Axis Attention for Medical Image Segmentation
Pakistan Institute of Engineering & Applied Sciences（巴基斯坦工程与应用科学学院）；PIEAS Artificial Intelligence Center (PAIC)（PIEAS人工智能中心）；Center for Mathematical Sciences（数学科学中心）
TA-RNN-Medical-Hybrid: A Time-Aware and Interpretable Framework for Mortality Risk Prediction
Department of Computer Engineering, SRC.,Islamic Azad university（计算机工程系，伊斯兰教Azad大学）；School of Computer Engineering, Iran university of Science and technology（计算机工程学院，伊朗科学技术大学）
Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach
Guangdong University of Technology（广东工业大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Guangzhou University of Traditional Chinese Medicine（广州中医药大学）；The Second Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第二附属医院）
Background and purpose: Heart disease has been one of the most important causes of death in the last 10 years, so the use of classification methods to diagnose and predict heart disease is very important. If this disease is predicted before menstruation, it is possible to prevent high mortality of the disease and provide more accurate and efficient treatment methods. Materials and Methods: Due to the selection of input features, the use of basic algorithms can be very time-consuming. Reducing dimensions or choosing a good subset of features, without risking accuracy, has great importance for b…
Prediction of medical codes from clinical notes is both a practical and essential need for every healthcare delivery organization within current medical systems. Automating annotation will save significant time and excessive effort spent by human coders today. However, the biggest challenge is directly identifying appropriate medical codes out of several thousands of high-dimensional codes from unstructured free-text clinical notes. In the past three years, with Convolutional Neural Networks (CNN) and Long Short-Term Memory (LTSM) networks, there have been vast improvements in tackling the mos…
Lancaster University（兰卡斯特大学）；Durham University（杜伦大学）；University of Central Arkansas（中阿肯色大学）；Queens University Belfast（贝尔法斯特女王大学）
Behavioural phenotyping of Drosophila is an important means in biological and medical research to identify genetic, pathologic or psychologic impact on animal behaviour.
Many systems based on knowledge, especially expert systems for medical decision support have been developed. Only systems are based on production rules, and cannot learn and evolve only by updating them. In addition, taking into account several criteria induces an exorbitant number of rules to be injected into the system. It becomes difficult to translate medical knowledge or a support decision as a simple rule. Moreover, reasoning based on generic cases became classic and can even reduce the range of possible solutions. To remedy that, we propose an approach based on using a multi-criteria de…
