Deep learning algorithms have seen acute growth of interest in their applications throughout several fields of interest in the last decade, with medical hyperspectral imaging being a particularly promising domain. So far, to the best of our knowledge, there is no review paper that discusses the implementation of deep learning for medical hyperspectral imaging, which is what this review paper aims to accomplish by examining publications that currently utilize deep learning to perform effective analysis of medical hyperspectral imagery. This paper discusses deep learning concepts that are relevant and applicable to medical hyperspectral imaging analysis, several of which have been implemented since the boom in deep learning. This will comprise of reviewing the use of deep learning for classification, segmentation, and detection in order to investigate the analysis of medical hyperspectral imaging. Lastly, we discuss the current and future challenges pertaining to this discipline and the possible efforts to overcome such trials.
Scribble-Based Interactive Segmentation of Medical Hyperspectral Images
School of Biomedical Engineering & Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）；Department of Neurosurgery, King’s College Hospital（国王学院医院神经外科）；Hypervision Surgical（Hypervision Surgical公司）
Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspec…
In-Vivo Hyperspectral Human Brain Image Database for Brain Cancer Detection
University of Las Palmas de Gran Canaria（大加那利岛拉斯帕尔马斯大学）；University Hospital Doctor Negrin（内格林大学医院）；University Hospital Southampton（南安普顿大学医院）；Addenbrookes Hospital（阿登布鲁克医院）；University of Cambridge（剑桥大学）；AKKA Technologies（AKKA Technologies公司）；Imperial College London（伦敦帝国理工学院）；Universidad Politécnica de Madrid（马德里理工大学）；Ecole Nationale Supérieure des Mines de Paris（巴黎高等矿业学校）；MINES ParisTech（巴黎矿业学院）
College of Optical and Electronic Technology, China Jiliang University（中国计量大学光学与电子科技学院）
Hyperspectral image (HSI) contains both spatial pattern and spectral information which has been widely used in food safety, remote sensing, and medical detection. However, the acquisition of hyperspectral images is usually costly due to the complicated apparatus for the acquisition of optical spectrum. Recently, it has been reported that HSI can be reconstructed from single RGB image using convolution neural network (CNN) algorithms. Compared with the traditional hyperspectral cameras, the method based on CNN algorithms is simple, portable and low cost. In this study, we focused on the influen…
University of California, Berkeley（加利福尼亚大学伯克利分校）；UCB/UCSF Joint Graduate Program in Bioengineering（加州大学伯克利分校/加州大学旧金山分校联合生物工程研究生项目）
Hyperspectral imaging is useful for applications ranging from medical diagnostics to agricultural crop monitoring; however, traditional scanning hyperspectral imagers are prohibitively slow and expensive for widespread adoption. Snapshot techniques exist but are often confined to bulky benchtop setups or have low spatio-spectral resolution. In this paper, we propose a novel, compact, and inexpensive computational camera for snapshot hyperspectral imaging. Our system consists of a tiled spectral filter array placed directly on the image sensor and a diffuser placed close to the sensor. Each poi…
Visual discrimination of clinical tissue types remains challenging, with traditional RGB imaging providing limited contrast for such tasks. Hyperspectral imaging (HSI) is a promising technology providing rich spectral information that can extend far beyond three-channel RGB imaging. Moreover, recently developed snapshot HSI cameras enable real-time imaging with significant potential for clinical applications. Despite this, the investigation into the relative performance of HSI over RGB imaging for semantic segmentation purposes has been limited, particularly in the context of medical imaging. …
Hamburg University of Technology（汉堡工业大学）；Carl von Ossietzky University Oldenburg（奥尔登堡大学）；University Medical Center Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心）；University of Bonn（波恩大学）；Klinikum Braunschweig（不伦瑞克医院）
Early detection of cancerous tissue is crucial for long-term patient survival. In the head and neck region, a typical diagnostic procedure is an endoscopic intervention where a medical expert manually assesses tissue using RGB camera images. While healthy and tumor regions are generally easier to distinguish, differentiating benign and malignant tumors is very challenging. This requires an invasive biopsy, followed by histological evaluation for diagnosis. Also, during tumor resection, tumor margins need to be verified by histological analysis. To avoid unnecessary tissue resection, a non-inva…
Advancements in Feature Extraction Recognition of Medical Imaging Systems Through Deep Learning Technique
Marquette University（马凯特大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；New York University（纽约大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of Southern California（南加州大学）；University of Houston（休斯顿大学）
Linear spectral unmixing is an essential technique in hyperspectral image processing and interpretation. In recent years, deep learning-based approaches have shown great promise in hyperspectral unmixing, in particular, unsupervised unmixing methods based on autoencoder networks are a recent trend. The autoencoder model, which automatically learns low-dimensional representations (abundances) and reconstructs data with their corresponding bases (endmembers), has achieved superior performance in hyperspectral unmixing. In this article, we explore the effective utilization of spatial and spectral…
Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations
University of Las Palmas de Gran Canaria（大加那利岛拉斯帕尔马斯大学）；Imperial College London（伦敦帝国理工学院）；University Hospital Doctor Negrin（内格林大学医院）；University Hospital Southampton（南安普顿大学医院）；University of Cambridge（剑桥大学）；Universidad Politecnica de Madrid（马德里理工大学）
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；International Research Center of Big Data for Sustainable Development Goals（可持续发展大数据国际研究中心）
Owing to the rapid development of sensor technology, hyperspectral (HS) remote sensing (RS) imaging has provided a significant amount of spatial and spectral information for the observation and analysis of the Earth's surface at a distance of data acquisition devices, such as aircraft, spacecraft, and satellite. The recent advancement and even revolution of the HS RS technique offer opportunities to realize the full potential of various applications, while confronting new challenges for efficiently processing and analyzing the enormous HS acquisition data. Due to the maintenance of the 3-D HS …
School of Biomedical Engineering, University of Science and Technology of China（中国科学技术大学生物医学工程学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tel-Aviv University（特拉维夫大学）；University of Pennsylvania（宾夕法尼亚大学）；Yale University（耶鲁大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Case Western Reserve University（凯斯西储大学）；Louis Stokes Cleveland Veterans Administration Medical Center（路易斯·斯托克斯克利夫兰退伍军人事务医疗中心）；Johns Hopkins University（约翰霍普金斯大学）；Klinikum rechts der Isar, TU Munich（慕尼黑工业大学右岸医院）；Imperial College（帝国理工学院）；National Institute of Health Clinical Center（美国国立卫生研究院临床中心）
Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs a…
From CNN to Transformer: A Review of Medical Image Segmentation Models
University of Electronic Science and Technology of China（电子科技大学）；Deyang People’s Hospital（德阳市人民医院）；Sichuan Provincial People’s Hospital（四川省人民医院）；Chinese Academy of Sciences Sichuan Translational Medicine Research Hospital（中国科学院四川转化医学研究医院）
Disentangling Progress in Medical Image Registration: Beyond Trend-Driven Architectures towards Domain-Specific Strategies
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Imperial College London（伦敦帝国理工学院）；University of Pennsylvania（宾夕法尼亚大学）；National University of Singapore（新加坡国立大学）
This paper discusses current methods and trends for 3D bounding box detection in volumetric medical image data. For this purpose, an overview of relevant papers from recent years is given. 2D and 3D implementations are discussed and compared. Multiple identified approaches for localizing anatomical structures are presented. The results show that most research recently focuses on Deep Learning methods, such as Convolutional Neural Networks vs. methods with manual feature engineering, e.g. Random-Regression-Forests. An overview of bounding box detection options is presented and helps researchers…
Uncertainty-Aware Multimodal Fusion for Oral Lesion Classification
1 School of Medical Science \& Technology, IIT Kharagpur, India 2 Department of Electrical Engineering, IIT Kharagpur, India 3 Dr. B.C. Roy Multispeciality Medical Research Centre, IIT Kharagpur, India
Purdue University Northwest（普渡大学西北校区）；Michigan Technological University（密歇根理工大学）
U-net is an image segmentation technique developed primarily for medical image analysis that can precisely segment images using a scarce amount of training data. These traits provide U-net with a very high utility within the medical imaging community and have resulted in extensive adoption of U-net as the primary tool for segmentation tasks in medical imaging. The success of U-net is evident in its widespread use in all major image modalities from CT scans and MRI to X-rays and microscopy. Furthermore, while U-net is largely a segmentation tool, there have been instances of the use of U-net in…
Emory University（埃默里大学）；Winship Cancer Institute, Emory University（埃默里大学温希普癌症研究所）
This paper presents a review of deep learning (DL) based medical image registration methods. We summarized the latest developments and applications of DL-based registration methods in the medical field. These methods were classified into seven categories according to their methods, functions and popularity. A detailed review of each category was presented, highlighting important contributions and identifying specific challenges. A short assessment was presented following the detailed review of each category to summarize its achievements and future potentials. We provided a comprehensive compar…
In this extended abstract, we will present and discuss opportunities and challenges brought about by a new deep learning method by AUC maximization (aka \underline{\bf D}eep \underline{\bf A}UC \underline{\bf M}aximization or {\bf DAM}) for medical image classification. Since AUC (aka area under ROC curve) is a standard performance measure for medical image classification, hence directly optimizing AUC could achieve a better performance for learning a deep neural network than minimizing a traditional loss function (e.g., cross-entropy loss). Recently, there emerges a trend of using deep AUC ma…
The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) …
Graz University of Technology（格拉茨工业大学）；Medical University of Graz（格拉茨医科大学）；University Medicine Essen（埃森大学医学院）；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））
Deep learning (DL) has remarkably impacted several different scientific disciplines over the last few years. E.g., in image processing and analysis, DL algorithms were able to outperform other cutting-edge methods. Additionally, DL has delivered state-of-the-art results in tasks like autonomous driving, outclassing previous attempts. There are even instances where DL outperformed humans, for example with object recognition and gaming. DL is also showing vast potential in the medical domain. With the collection of large quantities of patient records and data, and a trend towards personalized tr…
Detection of colon polyps has become a trending topic in the intersecting fields of machine learning and gastrointestinal endoscopy. The focus has mainly been on per-frame classification. More recently, polyp segmentation has gained attention in the medical community. Segmentation has the advantage of being more accurate than per-frame classification or object detection as it can show the affected area in greater detail. For our contribution to the EndoCV 2021 segmentation challenge, we propose two separate approaches. First, a segmentation model named TriUNet composed of three separate UNet m…
Shiraz University of Technology（设拉子理工大学）；Shiraz University of Medical Science（设拉子医科大学）
Image-guided interventions are saving the lives of a large number of patients where the image registration problem should indeed be considered as the most complex and complicated issue to be tackled. On the other hand, the recently huge progress in the field of machine learning made by the possibility of implementing deep neural networks on the contemporary many-core GPUs opened up a promising window to challenge with many medical applications, where the registration is not an exception. In this paper, a comprehensive review on the state-of-the-art literature known as medical image registratio…
National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”（乌克兰国立技术大学“伊戈尔·西科尔斯基基辅理工学院”）
The coronavirus pandemic has been going on since the year 2019, and the trend is still not abating. Therefore, it is particularly important to classify medical CT scans to assist in medical diagnosis. At present, Supervised Deep Learning algorithms have made a great success in the classification task of medical CT scans, but medical image datasets often require professional image annotation, and many research datasets are not publicly available. To solve this problem, this paper is inspired by the self-supervised learning algorithm MAE and uses the MAE model pre-trained on ImageNet to perform …
Medical imaging is crucial in modern clinics to guide the diagnosis and treatment of diseases. Medical image reconstruction is one of the most fundamental and important components of medical imaging, whose major objective is to acquire high-quality medical images for clinical usage at minimal cost and risk to the patients. Mathematical models in medical image reconstruction or, more generally, image restoration in computer vision, have been playing a prominent role. Earlier mathematical models are mostly designed by human knowledge or hypothesis on the image to be reconstructed, and we shall c…
School of Electrical Engineering, Tel Aviv University（特拉维夫大学电气工程学院）
Medical image analysis studies usually require medical image datasets for training, testing and validation of algorithms. The need is underscored by the deep learning revolution and the dominance of machine learning in recent medical image analysis research. Nevertheless, due to ethical and legal constraints, commercial conflicts and the dependence on busy medical professionals, medical image analysis researchers have been described as "data starved". Due to the lack of objective criteria for sufficiency of dataset size, the research community implicitly sets ad-hoc standards by means of the p…
MBZ University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；Monash University（蒙纳士大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Agency for Science, Technology and Research（新加坡科技研究局）
Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as {de facto} operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of…
The application of machine learning to radiological images is an increasingly active research area that is expected to grow in the next five to ten years. Recent advances in machine learning have the potential to recognize and classify complex patterns from different radiological imaging modalities such as x-rays, computed tomography, magnetic resonance imaging and positron emission tomography imaging. In many applications, machine learning based systems have shown comparable performance to human decision-making. The applications of machine learning are the key ingredients of future clinical d…
With the proliferation of inexpensive CMOS cameras, medical imaging experiences a noticeable influx of new technologies. While anatomical imaging is based on well-established principles of photography, physiological optical imaging is a relatively novel range of technologies that requires considering a new set of technical and physiological aspects. We discuss several factors (binning, spatial frequency sampling, and distance to the target area) indispensable to getting quantifiable and reproducible results, which are the essence of physiological imaging. We also discussed their implications f…
Is the medical image segmentation problem solved? A survey of current developments and future directions
University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；University of Pennsylvania（宾夕法尼亚大学）；Cleveland Clinic（克利夫兰诊所）；Mayo Clinic（梅奥诊所）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）
