Motivation: Deep learning models deployed for use on medical tasks can be equipped with Out-of-Distribution Detection (OoDD) methods in order to avoid erroneous predictions. However it is unclear which OoDD method should be used in practice. Specific Problem: Systems trained for one particular domain of images cannot be expected to perform accurately on images of a different domain. These images should be flagged by an OoDD method prior to diagnosis. Our approach: This paper defines 3 categories of OoD examples and benchmarks popular OoDD methods in three domains of medical imaging: chest X-ray, fundus imaging, and histology slides. Results: Our experiments show that despite methods yielding good results on some categories of out-of-distribution samples, they fail to recognize images close to the training distribution. Conclusion: We find a simple binary classifier on the feature representation has the best accuracy and AUPRC on average. Users of diagnostic tools which employ these OoDD methods should still remain vigilant that images very close to the training distribution yet not in it could yield unexpected results.
Medical imaging AI systems such as disease classification and segmentation are increasingly inspired and transformed from computer vision based AI systems. Although an array of adversarial training and/or loss function based defense techniques have been developed and proved to be effective in computer vision, defending against adversarial attacks on medical images remains largely an uncharted territory due to the following unique challenges: 1) label scarcity in medical images significantly limits adversarial generalizability of the AI system; 2) vastly similar and dominant fore- and backgroun…
Detection of Out-of-Distribution (OOD) samples in real time is a crucial safety check for deployment of machine learning models in the medical field. Despite a growing number of uncertainty quantification techniques, there is a lack of evaluation guidelines on how to select OOD detection methods in practice. This gap impedes implementation of OOD detection methods for real-world applications. Here, we propose a series of practical considerations and tests to choose the best OOD detector for a specific medical dataset. These guidelines are illustrated on a real-life use case of Electronic Healt…
Manipal Institute of Technology（马尼帕尔理工学院）；University of Groningen（格罗宁根大学）
Trusting the predictions of deep learning models in safety critical settings such as the medical domain is still not a viable option. Distentangled uncertainty quantification in the field of medical imaging has received little attention. In this paper, we study disentangled uncertainties in image to image translation tasks in the medical domain. We compare multiple uncertainty quantification methods, namely Ensembles, Flipout, Dropout, and DropConnect, while using CycleGAN to convert T1-weighted brain MRI scans to T2-weighted brain MRI scans. We further evaluate uncertainty behavior in the pre…
When deploying machine learning models in high-stakes real-world environments such as health care, it is crucial to accurately assess the uncertainty concerning a model's prediction on abnormal inputs. However, there is a scarcity of literature analyzing this problem on medical data, especially on mixed-type tabular data such as Electronic Health Records. We close this gap by presenting a series of tests including a large variety of contemporary uncertainty estimation techniques, in order to determine whether they are able to identify out-of-distribution (OOD) patients. In contrast to previous…
African Institute for Mathematical Sciences（非洲数学科学研究所）；Stanford University（斯坦福大学）
Deep learning models have become a popular choice for medical image analysis. However, the poor generalization performance of deep learning models limits them from being deployed in the real world as robustness is critical for medical applications. For instance, the state-of-the-art Convolutional Neural Networks (CNNs) fail to detect adversarial samples or samples drawn statistically far away from the training distribution. In this work, we experimentally evaluate the robustness of a Mahalanobis distance-based confidence score, a simple yet effective method for detecting abnormal input samples…
Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detection
Peng Cheng Laboratory（鹏城实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Nottingham Malaysia（诺丁汉大学（马来西亚））；Key Laboratory of Machine Intelligence and Advanced Computing, MOE（机器智能与高级计算重点实验室）
Machine learning-based medical anomaly detection is an important problem that has been extensively studied. Numerous approaches have been proposed across various medical application domains and we observe several similarities across these distinct applications. Despite this comparability, we observe a lack of structured organisation of these diverse research applications such that their advantages and limitations can be studied. The principal aim of this survey is to provide a thorough theoretical analysis of popular deep learning techniques in medical anomaly detection. In particular, we cont…
Federal Institute of Education, Science and Technology of Brasilia（巴西利亚联邦教育、科学与技术学院）
Telemedicine and mobile health applications, especially during the quarantine imposed by the covid-19 pandemic, led to an increase on the need of transferring health monitor readings from patients to specialists. Considering that most home medical devices use seven-segment displays, an automatic display reading algorithm should provide a more reliable tool for remote health care. This work proposes an end-to-end method for detection and reading seven-segment displays from medical devices based on deep learning object detection models. Two state of the art model families, EfficientDet and Effic…
Obtaining ground truth data in medical imaging has difficulties due to the fact that it requires a lot of annotating time from the experts in the field. Also, when trained with supervised learning, it detects only the cases included in the labels. In real practice, we want to also open to other possibilities than the named cases while examining the medical images. As a solution, the need for anomaly detection that can detect and localize abnormalities by learning the normal characteristics using only normal images is emerging. With medical image data, we can design either 2D or 3D networks of …
We present MedMNIST, a collection of 10 pre-processed medical open datasets. MedMNIST is standardized to perform classification tasks on lightweight 28x28 images, which requires no background knowledge. Covering the primary data modalities in medical image analysis, it is diverse on data scale (from 100 to 100,000) and tasks (binary/multi-class, ordinal regression and multi-label). MedMNIST could be used for educational purpose, rapid prototyping, multi-modal machine learning or AutoML in medical image analysis. Moreover, MedMNIST Classification Decathlon is designed to benchmark AutoML algori…
Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology(Shenzhen)（哈尔滨工业大学（深圳））；National University of Defense Technology（国防科技大学）；Peking University（北京大学）
In this paper, we present OpenMedIA, an open-source toolbox library containing a rich set of deep learning methods for medical image analysis under heterogeneous Artificial Intelligence (AI) computing platforms. Various medical image analysis methods, including 2D/3D medical image classification, segmentation, localisation, and detection, have been included in the toolbox with PyTorch and/or MindSpore implementations under heterogeneous NVIDIA and Huawei Ascend computing systems. To our best knowledge, OpenMedIA is the first open-source algorithm library providing compared PyTorch and MindSpor…
This paper discusses current methods and trends for 3D bounding box detection in volumetric medical image data. For this purpose, an overview of relevant papers from recent years is given. 2D and 3D implementations are discussed and compared. Multiple identified approaches for localizing anatomical structures are presented. The results show that most research recently focuses on Deep Learning methods, such as Convolutional Neural Networks vs. methods with manual feature engineering, e.g. Random-Regression-Forests. An overview of bounding box detection options is presented and helps researchers…
Benchmark Evaluation of Federated Learning on Multi-organ Images
Hunan Provincial Key Laboratory on Bioinformatics, School of Computer Science and Engineering, Central South University（湖南生物信息省重点实验室，计算机科学与工程学院，中南大学）；Xinjiang Engineering Research Center of Big Data and Intelligent Software, School of software, Xinjiang University（新疆大数据与智能软件工程研究中心，软件学院，新疆大学）
Northwood High School（诺斯伍德高中）；Irvine Unified School District（尔湾联合学区）；Fowler School of Engineering（福勒工程学院）；Chapman University（查普曼大学）
Deep generative networks in recent years have reinforced the need for caution while consuming various modalities of digital information. One avenue of deepfake creation is aligned with injection and removal of tumors from medical scans. Failure to detect medical deepfakes can lead to large setbacks on hospital resources or even loss of life. This paper attempts to address the detection of such attacks with a structured case study. Specifically, we evaluate eight different machine learning algorithms, which including three conventional machine learning methods, support vector machine, random fo…
ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding
Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）；National Strategic Technology Research Institute, Seoul National University Hospital（首尔国立大学医院国家战略技术研究所）；Department of Medicine, Division of Gastroenterology, Massachusetts General Hospital（麻省总医院内科与消化内科部门）；Department of Neuroradiology, Heidelberg University Hospital（海德堡大学医院神经放射科部门）；King Abdulaziz Medical City, National Guard Health Affairs（国王阿卜杜勒-阿齐兹医疗城，国民守卫健康事务局）；Division of Pulmonary, Critical Care, and Sleep Medicine, University of Cincinnati（辛辛那提大学肺科、重症医学及睡眠医学部门）
The University of Tokyo Hospital（东京大学医院）；The University of Tokyo（东京大学）；RIKEN（理化学研究所）
Oversight in medical images is a crucial problem, and timely reporting of medical images is desired. Therefore, an all-purpose anomaly detection method that can detect virtually all types of lesions/diseases in a given image is strongly desired. However, few commercially available and versatile anomaly detection methods for medical images have been provided so far. Recently, anomaly detection methods built upon deep learning methods have been rapidly growing in popularity, and these methods seem to provide reasonable solutions to the problem. However, the workload to label the images necessary…
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
Stanford University（斯坦福大学）；Shenzhen University（深圳大学）；The Chinese University of Hong Kong（香港中文大学）；City University of Hong Kong（香港城市大学）；Renmin University of China（中国人民大学）
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Karolinska Institutet（卡罗林斯卡学院）；Sahlgrenska University Hospital（萨尔格伦斯卡大学医院）
Accurate and timely detection of medical adverse events (AEs) from free-text medical narratives is challenging. Natural language processing (NLP) with deep learning has already shown great potential for analyzing free-text data, but its application for medical AE detection has been limited. In this study we proposed deep learning based NLP (DL-NLP) models for efficient and accurate hip dislocation AE detection following total hip replacement from standard (radiology notes) and non-standard (follow-up telephone notes) free-text medical narratives. We benchmarked these proposed models with a wid…
Rethinking Foundation Models for Medical Image Classification through a Benchmark Study on MedMNIST
University of Oxford（牛津大学）；Nuffield Department of Population Health, University of Oxford（牛津大学纳菲尔德人口健康系）；Big Data Institute, University of Oxford（牛津大学大数据研究所）
Beijing Institute of Technology（北京理工大学）；Renmin University of China（中国人民大学）；University of Minnesota（明尼苏达大学）
Deep learning methods have contributed substantially to the rapid advancement of medical image segmentation, the quality of which relies on the suitable design of loss functions. Popular loss functions, including the cross-entropy and dice losses, often fall short of boundary detection, thereby limiting high-resolution downstream applications such as automated diagnoses and procedures. We developed a novel loss function that is tailored to reflect the boundary information to enhance the boundary detection. As the contrast between segmentation and background regions along the classification bou…
Towards Universal Unsupervised Anomaly Detection in Medical Imaging
Technical University of Munich（慕尼黑工业大学）；Helmholtz AI and Helmholtz Center Munich（亥姆霍兹人工智能与亥姆霍兹慕尼黑中心）；Klinikum Rechts der Isar（伊萨尔河右岸医院）；Imperial College London（伦敦帝国理工学院）
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；PAII Inc.（PAII公司）
Box representation has been extensively used for object detection in computer vision. Such representation is efficacious but not necessarily optimized for biomedical objects (e.g., glomeruli), which play an essential role in renal pathology. In this paper, we propose a simple circle representation for medical object detection and introduce CircleNet, an anchor-free detection framework. Compared with the conventional bounding box representation, the proposed bounding circle representation innovates in three-fold: (1) it is optimized for ball-shaped biomedical objects; (2) The circle representat…
SimulaMet（西穆拉大都会研究所）；UIT The Artic University of Norway（挪威UIT北极大学）；Sahlgrenska University Hospital-Mölndal（萨尔格伦斯卡大学医院-默恩达尔）；Swedish Medical Group（瑞典医疗集团）；Oslo Metropolitan University（奥斯陆城市大学）；Norwegian University of Science and Technology（挪威科技大学）
Deep learning has in recent years achieved immense success in all areas of computer vision and has the potential of assisting medical doctors in analyzing visual content for disease and other abnormalities. However, the current state of deep learning is very much a black box, making medical professionals highly skeptical about integrating these methods into clinical practice. Several methods have been proposed in order to shine some light onto these black boxes, but there is no consensus on the opinion of the medical doctors that will consume these explanations. This paper presents a study ask…
Microsoft Health and Life Sciences（微软健康与生命科学部）；Stanford Center for Artificial Intelligence in Medicine and Imaging（斯坦福医学与成像人工智能中心）
Clinical Artificial lntelligence (AI) applications are rapidly expanding worldwide, and have the potential to impact to all areas of medical practice. Medical imaging applications constitute a vast majority of approved clinical AI applications. Though healthcare systems are eager to adopt AI solutions a fundamental question remains: \textit{what happens after the AI model goes into production?} We use the CheXpert and PadChest public datasets to build and test a medical imaging AI drift monitoring workflow to track data and model drift without contemporaneous ground truth. We simulate drift in…
Label Augmentation Method for Medical Landmark Detection in Hip Radiograph Images
Vanderbilt University（范德堡大学）；Vanderbilt Institute for Surgery and Engineering（范德堡外科与工程研究所）；University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；Vanderbilt University Medical Center（范德堡大学医学中心）
本文提出仅增强标签的两阶段课程训练方法，通过扩大并逐步侵蚀标签区域训练 U-Net，在六个专家标注的髋关节 X 线数据集上实现优于传统数据增强的高效医学标志点检测。
University of Massachusetts（马萨诸塞大学）；Bedford VAMC and CHOIR（贝德福德退伍军人医疗中心与CHOIR）
Sequence labeling for extraction of medical events and their attributes from unstructured text in Electronic Health Record (EHR) notes is a key step towards semantic understanding of EHRs. It has important applications in health informatics including pharmacovigilance and drug surveillance. The state of the art supervised machine learning models in this domain are based on Conditional Random Fields (CRFs) with features calculated from fixed context windows. In this application, we explored various recurrent neural network frameworks and show that they significantly outperformed the CRF models.
A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning
Tokyo University of Agriculture and Technology（东京农业大学）；Institute of Science Tokyo（东京科学研究院）；National University of Singapore（国立新加坡大学）；LY Corporation（LY公司）；Altos Labs, Inc.（Altos实验室）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Emory University（埃默里大学）；Norwegian University of Science and Technology（挪威科学技术大学）
The paper describes the open Russian medical language understanding benchmark covering several task types (classification, question answering, natural language inference, named entity recognition) on a number of novel text sets. Given the sensitive nature of the data in healthcare, such a benchmark partially closes the problem of Russian medical dataset absence. We prepare the unified format labeling, data split, and evaluation metrics for new tasks. The remaining tasks are from existing datasets with a few modifications. A single-number metric expresses a model's ability to cope with the benc…
