We present in this paper a new approach for the automatic annotation of medical images, using the approach of "bag-of-words" to represent the visual content of the medical image combined with text descriptors based approach tf.idf and reduced by latent semantic to extract the co-occurrence between terms and visual terms. A medical report is composed of a text describing a medical image. First, we are interested to index the text and extract all relevant terms using a thesaurus containing MeSH medical concepts. In a second phase, the medical image is indexed while recovering areas of interest which are invariant to change in scale, light and tilt. To annotate a new medical image, we use the approach of "bagof-words" to recover the feature vector. Indeed, we use the vector space model to retrieve similar medical image from the database training. The calculation of the relevance value of an image to the query image is based on the cosine function. We conclude with an experiment carried out on five types of radiological imaging to evaluate the performance of our system of medical annotation. The results showed that our approach works better with more images from the radiology of the skull.
Medical images annotation is most of the time a repetitive hard task. Collecting old similar annotations and assigning them to new medical images may not only enhance the annotation process, but also reduce ambiguity caused by repetitive annotations. The goal of this work is to propose an approach based on Hausdorff distance able to compute similarity between a new medical image and old stored images. User has to choose then one of the similar images and annotations related to the selected one are assigned to the new one.
Automatically classifying the tissues types of Region of Interest (ROI) in medical imaging has been an important application in Computer-Aided Diagnosis (CAD), such as classification of breast parenchymal tissue in the mammogram, classify lung disease patterns in High-Resolution Computed Tomography (HRCT) etc. Recently, bag-of-features method has shown its power in this field, treating each ROI as a set of local features. In this paper, we investigate using the bag-of-features strategy to classify the tissue types in medical imaging applications. Two important issues are considered here: the v…
University of Science and Technology of China（中国科学技术大学）
Multi-Instance Learning(MIL) aims to learn the mapping between a bag of instances and the bag-level label. Therefore, the relationships among instances are very important for learning the mapping. In this paper, we propose an MIL algorithm based on a graph built by structural relationship among instances within a bag. Then, Graph Convolutional Network(GCN) and the graph-attention mechanism are used to learn bag-embedding. In the task of medical image classification, our GCN-based MIL algorithm makes full use of the structural relationships among patches(instances) in an original image space do…
Imaging has occupied a huge role in the management of patients, whether hospitalized or not. Depending on the patients clinical problem, a variety of imaging modalities were available for use. This gave birth of the annotation of medical image process. The annotation is intended to image analysis and solve the problem of semantic gap. The reason for image annotation is due to increase in acquisition of images. Physicians and radiologists feel better while using annotation techniques for faster remedy in surgery and medicine due to the following reasons: giving details to the patients, searchin…
ProMIL: Probabilistic Multiple Instance Learning for Medical Imaging
Faculty of Mathematics and Computer Science, Jagiellonian University（雅盖隆大学数学与计算机科学学院）；Faculty of Applied Computer Science, University of Information Technology and Management in Rzeszow（热舒夫信息技术与管理大学应用计算机科学学院）；Medical University of Gdansk（格但斯克医科大学）；UES Ltd.（UES有限公司）
Imperial College London（伦敦帝国学院）；Aichi Cancer Center（爱知县癌症中心）；Nagoya University（名古屋大学）
To efficiently establish training databases for machine learning methods, collaborative and crowdsourcing platforms have been investigated to collectively tackle the annotation effort. However, when this concept is ported to the medical imaging domain, reading expertise will have a direct impact on the annotation accuracy. In this study, we examine the impact of expertise and the amount of available annotations on the accuracy outcome of a liver segmentation problem in an abdominal computed tomography (CT) image database. In controlled experiments, we study this impact for different types of w…
Probabilistic smooth attention for deep multiple instance learning in medical imaging
Department of Computer Science and AI, University of Granada（计算机科学与人工智能系，格拉纳达大学）；Department of Statistics and Operations Research, University of Granada（统计与运筹学系，格拉纳达大学）；Research Centre for Information and Communication Technologies (CITIC), University of Granada（信息与通信技术研究中心（CITIC），格拉纳达大学）；Department of Electrical and Computer Engineering, Northwestern University（电气与计算机工程系，西北大学）；Center for Computational Imaging and Signal Analytics in Medicine, Northwestern University（医学计算成像与信号分析中心，西北大学）
Interactive Medical-SAM2 GUI: A Napari-based semi-automatic annotation tool for medical images
Department of Biomechatronic Engineering, Sungkyunkwan University（生物机电工程系，全州大学）；Department of Neurosurgery, Seoul National University Hospital, Seoul National University College of Medicine（神经外科，首尔国立大学医院，首尔国立大学医学院）
ETH Zürich（苏黎世联邦理工学院）；University of Cambridge（剑桥大学）
Semantic segmentation of medical images is a crucial step for the quantification of healthy anatomy and diseases alike. The majority of the current state-of-the-art segmentation algorithms are based on deep neural networks and rely on large datasets with full pixel-wise annotations. Producing such annotations can often only be done by medical professionals and requires large amounts of valuable time. Training a medical image segmentation network with weak annotations remains a relatively unexplored topic. In this work we investigate training strategies to learn the parameters of a pixel-wise s…
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）；West China Hospital, Sichuan University（四川大学华西医院）
Background and Objective: Open-source deep learning toolkits are one of the driving forces for developing medical image segmentation models. Existing toolkits mainly focus on fully supervised segmentation and require full and accurate pixel-level annotations that are time-consuming and difficult to acquire for segmentation tasks, which makes learning from imperfect labels highly desired for reducing the annotation cost. We aim to develop a new deep learning toolkit to support annotation-efficient learning for medical image segmentation. Methods: Our proposed toolkit named PyMIC is a modular de…
Image Sciences Institute, University Medical Center Utrecht（乌得勒支大学医学中心图像科学研究所）；Eindhoven University of Technology（埃因霍温理工大学）
Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annot…
Sunnybrook Research Institute（桑尼布鲁克研究所）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
To alleviate the burden of gathering detailed expert annotations when training deep neural networks, we propose a weakly supervised learning approach to recognize metastases in microscopic images of breast lymph nodes. We describe an alternative training loss which clusters weakly labeled bags in latent space to inform relevance of patch-instances during training of a convolutional neural network. We evaluate our method on the Camelyon dataset which contains high-resolution digital slides of breast lymph nodes, where labels are provided at the image-level and only subsets of patches are made a…
Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；Massachusetts Institute of Technology（麻省理工学院）
Ranking by pairwise comparisons has shown improved reliability over ordinal classification. However, as the annotations of pairwise comparisons scale quadratically, this becomes less practical when the dataset is large. We propose a method for reducing the number of pairwise comparisons required to rank by a quantitative metric, demonstrating the effectiveness of the approach in ranking medical images by image quality in this proof of concept study. Using the medical image annotation software that we developed, we actively subsample pairwise comparisons using a sorting algorithm with a human r…
DExTeR: Weakly Semi-Supervised Object Detection with Class and Instance Experts for Medical Imaging
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、法国）；University Hospital of Strasbourg, France（斯特拉斯堡大学医院、法国）
Though analysis of Medical Images by Deep Learning achieves unprecedented results across various applications, the effect of \emph{noisy training annotations} is rarely studied in a systematic manner. In Medical Image Analysis, most reports addressing this question concentrate on studying segmentation performance of deep learning classifiers. The absence of continuous ground truth annotations in these studies limits the value of conclusions for applications, where regression is the primary method of choice. In the application of surgical instrument pose estimation, where precision has a direct…
VinBigData JSC（VinBigData股份公司）；VinUniversity（VinUniversity（温纳大学））；VinUni-Illinois Smart Health Center（VinUni-伊利诺伊智慧健康中心）
Building an accurate computer-aided diagnosis system based on data-driven approaches requires a large amount of high-quality labeled data. In medical imaging analysis, multiple expert annotators often produce subjective estimates about "ground truth labels" during the annotation process, depending on their expertise and experience. As a result, the labeled data may contain a variety of human biases with a high rate of disagreement among annotators, which significantly affect the performance of supervised machine learning algorithms. To tackle this challenge, we propose a simple yet effective a…
Tsinghua University（清华大学）；Peking University Shenzhen Hospital（北京大学深圳医院）
Most of the state-of-the-art semantic segmentation reported in recent years is based on fully supervised deep learning in the medical domain. How?ever, the high-quality annotated datasets require intense labor and domain knowledge, consuming enormous time and cost. Previous works that adopt semi?supervised and unsupervised learning are proposed to address the lack of anno?tated data through assisted training with unlabeled data and achieve good perfor?mance. Still, these methods can not directly get the image annotation as doctors do. In this paper, inspired by self-training of semi-supervised…
Learning Robust Medical Image Segmentation from Multi-source Annotations
Tsinghua University（清华大学）；Hong Kong University of Science and Technology（香港科技大学）；Chinese Academy of Sciences（中国科学院）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）
Image annotation is one essential prior step to enable data-driven algorithms. In medical imaging, having large and reliably annotated data sets is crucial to recognize various diseases robustly. However, annotator performance varies immensely, thus impacts model training. Therefore, often multiple annotators should be employed, which is however expensive and resource-intensive. Hence, it is desirable that users should annotate unseen data and have an automated system to unobtrusively rate their performance during this process. We examine such a system based on whole slide images (WSIs) showin…
University of Calgary（卡尔加里大学）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (AMII)（阿尔伯塔机器智能研究所（AMII））
Many successful methods developed for medical image analysis that are based on machine learning use supervised learning approaches, which often require large datasets annotated by experts to achieve high accuracy. However, medical data annotation is time-consuming and expensive, especially for segmentation tasks. To solve the problem of learning with limited labeled medical image data, an alternative deep learning training strategy based on self-supervised pretraining on unlabeled MRI scans is proposed in this work. Our pretraining approach first, randomly applies different distortions to rand…
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）
Reducing outcome variance is an essential task in deep learning based medical image analysis. Bootstrap aggregating, also known as bagging, is a canonical ensemble algorithm for aggregating weak learners to become a strong learner. Random forest is one of the most powerful machine learning algorithms before deep learning era, whose superior performance is driven by fitting bagged decision trees (weak learners). Inspired by the random forest technique, we propose a simple bagging ensemble deep segmentation (BEDs) method to train multiple U-Nets with partial training data to segment dense nuclei…
National University of Singapore（新加坡国立大学）；National University Hospital（国立大学医院）；Zhejiang University（浙江大学）
In this paper, we propose a novel semi-supervised learning (SSL) framework named BoostMIS that combines adaptive pseudo labeling and informative active annotation to unleash the potential of medical image SSL models: (1) BoostMIS can adaptively leverage the cluster assumption and consistency regularization of the unlabeled data according to the current learning status. This strategy can adaptively generate one-hot "hard" labels converted from task model predictions for better task model training. (2) For the unselected unlabeled images with low confidence, we introduce an Active learning (AL) …
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）
Despite that deep learning has achieved state-of-the-art performance for medical image segmentation, its success relies on a large set of manually annotated images for training that are expensive to acquire. In this paper, we propose an annotation-efficient learning framework for segmentation tasks that avoids annotations of training images, where we use an improved Cycle-Consistent Generative Adversarial Network (GAN) to learn from a set of unpaired medical images and auxiliary masks obtained either from a shape model or public datasets. We first use the GAN to generate pseudo labels for our …
École de technologie supérieure, Université du Québec（魁北克大学高等技术学院）
A growing number of applications, e.g. video surveillance and medical image analysis, require training recognition systems from large amounts of weakly annotated data while some targeted interactions with a domain expert are allowed to improve the training process. In such cases, active learning (AL) can reduce labeling costs for training a classifier by querying the expert to provide the labels of most informative instances. This paper focuses on AL methods for instance classification problems in multiple instance learning (MIL), where data is arranged into sets, called bags, that are weakly …
MSE-Nets: Multi-annotated Semi-supervised Ensemble Networks for Improving Segmentation of Medical Image with Ambiguous Boundaries
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang Sci-Tech University（浙江理工大学）；Shandong University（山东大学）；Lishui University（丽水学院）；Center（研究中心）；Communication University of Zhejiang（浙江传媒学院）
Skip and Skip: Segmenting Medical Images with Prompts
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Engineering Research Center of AI and Robotics, Ministry of Education, China（教育部人工智能与机器人工程研究中心）；Jilin Provincial Key Laboratory of Intelligence Science（吉林省智能科学重点实验室）
Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Recent advances in deep learning have resulted in great successes in various applications. Although semi-supervised or unsupervised learning methods have been widely investigated, the performance of deep neural networks highly depends on the annotated data. The problem is that the budget for annotation is usually limited due to the annotation time and expensive annotation cost in medical data. Active learning is one of the solutions to this problem where an active learner is designed to indicate which samples need to be annotated to effectively train a target model. In this paper, we propose a…
SUNY Buffalo（纽约州立大学布法罗分校）；Medical College of Wisconsin（威斯康星医学院）；Washington University School of Medicine（华盛顿大学医学院）；University of California, Davis Medical Center（加州大学戴维斯医学中心）
Neural networks promise to bring robust, quantitative analysis to medical fields, but adoption is limited by the technicalities of training these networks. To address this translation gap between medical researchers and neural networks in the field of pathology, we have created an intuitive interface which utilizes the commonly used whole slide image (WSI) viewer, Aperio ImageScope (Leica Biosystems Imaging, Inc.), for the annotation and display of neural network predictions on WSIs. Leveraging this, we propose the use of a human-in-the-loop strategy to reduce the burden of WSI annotation. We …
Medical image segmentation requires consensus ground truth segmentations to be derived from multiple expert annotations. A novel approach is proposed that obtains consensus segmentations from experts using graph cuts (GC) and semi supervised learning (SSL). Popular approaches use iterative Expectation Maximization (EM) to estimate the final annotation and quantify annotator's performance. Such techniques pose the risk of getting trapped in local minima. We propose a self consistency (SC) score to quantify annotator consistency using low level image features. SSL is used to predict missing anno…
Computer Vision Lab, ETH Zurich（计算机视觉实验室，苏黎世联邦理工学院）
A key requirement for the success of supervised deep learning is a large labeled dataset - a condition that is difficult to meet in medical image analysis. Self-supervised learning (SSL) can help in this regard by providing a strategy to pre-train a neural network with unlabeled data, followed by fine-tuning for a downstream task with limited annotations. Contrastive learning, a particular variant of SSL, is a powerful technique for learning image-level representations. In this work, we propose strategies for extending the contrastive learning framework for segmentation of volumetric medical i…
Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation
Wuhan University of Technology（武汉理工大学）；Huazhong University of Science and Technology（华中科技大学）；Changzhou United lmaging Healthcare Surgical Technology Co. Ltd.（常州联影医疗手术技术有限公司）
提出Mask to Concept (M2C)框架，无需外部模块或重训练，仅用少量标注图像，通过可学习概念嵌入搜索和混合不确定性估计，实现SAM3在医学图像中的自动少样本标注，达到SOTA性能。
Weakly Supervised Medical Image Segmentation With Soft Labels and Noise Robust Loss
University of Calgary（卡尔加里大学）；McCaig Institute for Bone and Joint Health（麦凯格骨骼与关节健康研究所）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (AMII)（阿尔伯塔机器智能研究所）
University of Augsburg（奥格斯堡大学）；University Hospital Augsburg（奥格斯堡大学医院）
Novel and high-performance medical image classification pipelines are heavily utilizing ensemble learning strategies. The idea of ensemble learning is to assemble diverse models or multiple predictions and, thus, boost prediction performance. However, it is still an open question to what extent as well as which ensemble learning strategies are beneficial in deep learning based medical image classification pipelines. In this work, we proposed a reproducible medical image classification pipeline for analyzing the performance impact of the following ensemble learning techniques: Augmenting, Stack…
All-in-one platform for AI R&D in medical imaging, encompassing data collection, selection, annotation, and pre-processing
Callisto Inc（卡利斯托公司）；The University of Tokyo（东京大学）；St. Marianna University School of Medicine（圣玛丽安娜医科大学医学院）；Harada Academy（原田学园）；St. Marianna University Yokohama Seibu Hospital（圣玛丽安娜大学横滨西部医院）
Building RadiologyNET: Unsupervised annotation of a large-scale multimodal medical database
University of Rijeka – Faculty of Engineering（里耶卡大学工程学院）；Center for Artificial Intelligence and Cybersecurity（人工智能与网络安全中心）；Medical University of Graz（格拉茨医科大学）
Arizona State University（亚利桑那州立大学）；Mayo Clinic（梅奥诊所）
The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious, laborious, and time consuming, but it also demands costly, specialty-oriented skills, which are not easily accessible. To dramatically reduce annotation cost, this paper presents a novel method to naturally integrate active learning and transfer learning (fine-tuning) into a si…
National Institute of Technology Karnataka（卡纳塔克邦国家理工学院）
Medical Image Retrieval is a challenging field in Visual information retrieval, due to the multi-dimensional and multi-modal context of the underlying content. Traditional models often fail to take the intrinsic characteristics of data into consideration, and have thus achieved limited accuracy when applied to medical images. The Bag of Visual Words (BoVW) is a technique that can be used to effectively represent intrinsic image features in vector space, so that applications like image classification and similar-image search can be optimized. In this paper, we present a MedIR approach based on …
Decision support tools that rely on supervised learning require large amounts of expert annotations. Using past radiological reports obtained from hospital archiving systems has many advantages as training data above manual single-class labels: they are expert annotations available in large quantities, covering a population-representative variety of pathologies, and they provide additional context to pathology diagnoses, such as anatomical location and severity. Learning to auto-generate such reports from images present many challenges such as the difficulty in representing and generating long…
The ability to quickly annotate medical imaging data plays a critical role in training deep learning frameworks for segmentation. Doing so for image volumes or video sequences is even more pressing as annotating these is particularly burdensome. To alleviate this problem, this work proposes a new method to efficiently segment medical imaging volumes or videos using point-wise annotations only. This allows annotations to be collected extremely quickly and remains applicable to numerous segmentation tasks. Our approach trains a deep learning model using an appropriate Positive/Unlabeled objectiv…
TV-SAM: Increasing Zero-Shot Segmentation Performance on Multimodal Medical Images Using GPT-4 Generated Descriptive Prompts Without Human Annotation
Sichuan University（四川大学）；Case Western Reserve University（凯斯西储大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Urgench State University（乌尔根奇国立大学）；Gansu Agricultural University（甘肃农业大学）
BoMD: Bag of Multi-label Descriptors for Noisy Chest X-ray Classification
Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Harvard Medical School, Harvard University（哈佛医学院）；Centre for Vision, Speech and Signal Processing, University of Surrey（萨里大学视觉、语音与信号处理中心）
