Superpixel Pre-Segmentation of HER2 Slides for Efficient Annotation
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学模式识别实验室）；Institute of Pathology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学埃尔朗根医院病理学研究所）；Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN)（埃尔朗根-欧洲大都会区综合癌症中心）；Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学生物医学工程人工智能系）；Department of Gynecology and Obstetrics, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡大学埃尔朗根医院妇产科）
Mathias Öttl, Jana Mönius, Christian Marzahl, Matthias Rübner, Carol I. Geppert, Arndt Hartmann, Matthias W. Beckmann, Peter Fasching, Andreas Maier, Ramona Erb… 展开作者
Mathias Öttl, Jana Mönius, Christian Marzahl, Matthias Rübner, Carol I. Geppert, Arndt Hartmann, Matthias W. Beckmann, Peter Fasching, Andreas Maier, Ramona Erber, Katharina Breininger
Supervised deep learning has shown state-of-the-art performance for medical image segmentation across different applications, including histopathology and cancer research; however, the manual annotation of such data is extremely laborious. In this work, we explore the use of superpixel approaches to compute a pre-segmentation of HER2 stained images for breast cancer diagnosis that facilitates faster manual annotation and correction in a second step. Four methods are compared: Standard Simple Linear Iterative Clustering (SLIC) as a baseline, a domain adapted SLIC, and superpixels based on feature embeddings of a pretrained ResNet-50 and a denoising autoencoder. To tackle oversegmentation, we propose to hierarchically merge superpixels, based on their content in the respective feature space. When evaluating the approaches on fully manually annotated images, we observe that the autoencoder-based superpixels achieve a 23% increase in boundary F1 score compared to the baseline SLIC superpixels. Furthermore, the boundary F1 score increases by 73% when hierarchical clustering is applied on the adapted SLIC and the autoencoder-based superpixels. These evaluations show encouraging first results for a pre-segmentation for efficient manual refinement without the need for an initial set of annotated training data.
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Freie Universität Berlin（柏林自由大学）
Large-scale image data such as digital whole-slide histology images pose a challenging task at annotation software solutions. Today, a number of good solutions with varying scopes exist. For cell annotation, however, we find that many do not match the prerequisites for fast annotations. Especially in the field of mitosis detection, it is assumed that detection accuracy could significantly benefit from larger annotation databases that are currently however very troublesome to produce. Further, multiple independent (blind) expert labels are a big asset for such databases, yet there is currently …
Enhancing Point Annotations with Superpixel and Confidence Learning Guided for Improving Semi-Supervised OCT Fluid Segmentation
Zhejiang Sci-Tech University（浙江理工大学）；Lishui University（丽水学院）；the Second Affiliated Hospital of Zhejiang University, College of Medicine（浙江大学医学院附属第二医院）；Communication University of Zhejiang（浙江传媒学院）；Hangzhou Dianzi University（杭州电子科技大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）；Suzhou Research Institute of Shandong University（山东大学苏州研究院）
Segmentation of objects of interest is one of the central tasks in medical image analysis, which is indispensable for quantitative analysis. When developing machine-learning based methods for automated segmentation, manual annotations are usually used as the ground truth toward which the models learn to mimic. While the bulky parts of the segmentation targets are relatively easy to label, the peripheral areas are often difficult to handle due to ambiguous boundaries and the partial volume effect, etc., and are likely to be labeled with uncertainty. This uncertainty in labeling may, in turn, re…
Annotating cancerous regions in whole-slide images (WSIs) of pathology samples plays a critical role in clinical diagnosis, biomedical research, and machine learning algorithms development. However, generating exhaustive and accurate annotations is labor-intensive, challenging, and costly. Drawing only coarse and approximate annotations is a much easier task, less costly, and it alleviates pathologists' workload. In this paper, we study the problem of refining these approximate annotations in digital pathology to obtain more accurate ones. Some previous works have explored obtaining machine le…
Bayer AG（拜耳股份公司）；Median Technologies（Median科技公司）；Bayer Crop Science（拜耳作物科学公司）；Pfizer Worldwide Research Development and Medical（辉瑞全球研发与医疗部门）
In pre-clinical pathology, there is a paradox between the abundance of raw data (whole slide images from many organs of many individual animals) and the lack of pixel-level slide annotations done by pathologists. Due to time constraints and requirements from regulatory authorities, diagnoses are instead stored as slide labels. Weakly supervised training is designed to take advantage of those data, and the trained models can be used by pathologists to rank slides by their probability of containing a given lesion of interest. In this work, we propose a novel contextualized eXplainable AI (XAI) f…
Obtaining a large amount of labeled data in medical imaging is laborious and time-consuming, especially for histopathology. However, it is much easier and cheaper to get unlabeled data from whole-slide images (WSIs). Semi-supervised learning (SSL) is an effective way to utilize unlabeled data and alleviate the need for labeled data. For this reason, we proposed a framework that employs an SSL method to accurately detect cancerous regions with a novel annotation method called Minimal Point-Based annotation, and then utilize the predicted results with an innovative hybrid loss to train a classif…
Slide-Level Active Learning Reduces Annotation Burden in H&E images
Institute for Biomedical Informatics, Faculty of Medicine（生物医学信息学研究所，医学学院）；University Hospital Cologne, University of Cologne（科隆大学医院，科隆大学）；Center for Molecular Medicine Cologne (CMMC), Faculty of Medicine（科隆分子医学中心（CMMC），医学学院）；Cologne Excellence Cluster on Cellular Stress Responses in Aging-Associated Diseases (CECAD), University of Cologne（科隆卓越集群：与衰老相关疾病细胞应激反应（CECAD），科隆大学）；Faculty of Mathematics（数学学院）；Natural Sciences, University of Cologne（自然科学学院，科隆大学）；Institute of Pathology, University Hospital Cologne（病理学研究所，科隆大学医院）
Uppsala University（乌普萨拉大学）；SciLifeLab（瑞典生命科学实验室）；Radboud University Medical Center（拉德堡德大学医学中心）
Annotations are necessary to develop computer vision algorithms for histopathology, but dense annotations at a high resolution are often time-consuming to make. Deep learning models for segmentation are a way to alleviate the process, but require large amounts of training data, training times and computing power. To address these issues, we present seeded iterative clustering to produce a coarse segmentation densely and at the whole slide level. The algorithm uses precomputed representations as the clustering space and a limited amount of sparse interactive annotations as seeds to iteratively …
Fraunhofer Institute for Integrated Circuits IIS（弗劳恩霍夫集成电路研究所IIS）；Friedrich-Alexander-University, Erlangen-Nuremberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；University Hospital Erlangen, FAU Erlangen-Nuremberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学埃尔朗根大学医院）
Automatic outlining of different tissue types in digitized histological specimen provides a basis for follow-up analyses and can potentially guide subsequent medical decisions. The immense size of whole-slide-images (WSI), however, poses a challenge in terms of computation time. In this regard, the analysis of non-overlapping patches outperforms pixelwise segmentation approaches, but still leaves room for optimization. Furthermore, the division into patches, regardless of the biological structures they contain, is a drawback due to the loss of local dependencies. We propose to subdivide the WS…
Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Ulsan College of Medicine（蔚山大学医学院）；Asan Medical Center（峨山医学中心）；Ajou University School of Medicine（亚洲大学医学院）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Deep learning has been widely used to analyze digitized hematoxylin and eosin (H&E)-stained histopathology whole slide images. Automated cancer segmentation using deep learning can be used to diagnose malignancy and to find novel morphological patterns to predict molecular subtypes. To train pixel-wise cancer segmentation models, manual annotation from pathologists is generally a bottleneck due to its time-consuming nature. In this paper, we propose Deep Interactive Learning with a pretrained segmentation model from a different cancer type to reduce manual annotation time. Instead of annotatin…
Improving Quality Control of Whole Slide Images by Explicit Artifact Augmentation
AGH University of Krakow（克拉科夫AGH科技大学）；University of Applied Sciences Western Switzerland (HES-SO)（瑞士西部应用科学大学（HES-SO））；Radboud University Medical Center（拉德堡德大学医学中心）；University of Padova（帕多瓦大学）；University of Geneva（日内瓦大学）
National University of Sciences and Technology (NUST)（巴基斯坦国立科技大学（NUST））；Riphah International University（里法国际大学）
Unavailability of large training datasets is a bottleneck that needs to be overcome to realize the true potential of deep learning in histopathology applications. Although slide digitization via whole slide imaging scanners has increased the speed of data acquisition, labeling of virtual slides requires a substantial time investment from pathologists. Eye gaze annotations have the potential to speed up the slide labeling process. This work explores the viability and timing comparisons of eye gaze labeling compared to conventional manual labeling for training object detectors. Challenges associ…
Paired Uterine Whole-Slide Images and Pathology Reports for Multimodal Computational Pathology
Institute of Pathology, Technical University of Munich（慕尼黑工业大学病理研究所）；Computer Aided Medical Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助医疗程序（CAMP））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Human Pathology, Juntendo University Graduate School of Medicine（顺天堂大学医学研究生院人体病理学部）；The Hong Kong University of Science and Technology（香港科技大学）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Dartmouth College（达特茅斯学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希契科克医疗中心）
Deep learning-based methods, such as the sliding window approach for cropped-image classification and heuristic aggregation for whole-slide inference, for analyzing histological patterns in high-resolution microscopy images have shown promising results. These approaches, however, require a laborious annotation process and are fragmented. This diagnostic study collected deidentified high-resolution histological images (N = 379) for training a new model composed of a convolutional neural network and a grid-based attention network, trainable without region-of-interest annotations. Histological im…
From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformer
University of Science and Technology Beijing（北京科技大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；Changhai Hospital（长海医院）；Peking University（北京大学）
Histopathological images provide rich information for disease diagnosis. Large numbers of histopathological images have been digitized into high resolution whole slide images, opening opportunities in developing computational image analysis tools to reduce pathologists' workload and potentially improve inter- and intra- observer agreement. Most previous work on whole slide image analysis has focused on classification or segmentation of small pre-selected regions-of-interest, which requires fine-grained annotation and is non-trivial to extend for large-scale whole slide analysis. In this paper,…
Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡弗里德里希-亚历山大大学（FAU））；EUROIMMUN Medizinische Labordiagnostika AG（欧蒙医学实验诊断股份公司）；Freie Universität Berlin（柏林自由大学）；Technische Hochschule Ingolstadt（英戈尔施塔特工业大学）
Asthma is a chronic inflammatory disorder of the lower respiratory tract and naturally occurs in humans and animals including horses. The annotation of an asthma microscopy whole slide image (WSI) is an extremely labour-intensive task due to the hundreds of thousands of cells per WSI. To overcome the limitation of annotating WSI incompletely, we developed a training pipeline which can train a deep learning-based object detection model with partially annotated WSIs and compensate class imbalances on the fly. With this approach we can freely sample from annotated WSIs areas and are not restricte…
Digital Environment Research Institute（数字环境研究所）；Queen Mary University of London（伦敦玛丽女王大学）；William Harvey Research Institute（威廉·哈维研究所）；The Hong Kong Polytechnic University（香港理工大学）
Deep learning methods are widely applied in digital pathology to address clinical challenges such as prognosis and diagnosis. As one of the most recent applications, deep models have also been used to extract molecular features from whole slide images. Although molecular tests carry rich information, they are often expensive, time-consuming, and require additional tissue to sample. In this paper, we propose tRNAsfomer, an attention-based topology that can learn both to predict the bulk RNA-seq from an image and represent the whole slide image of a glass slide simultaneously. The tRNAsfomer use…
SUNY Buffalo（纽约州立大学布法罗分校）；Medical College of Wisconsin（威斯康星医学院）；Washington University School of Medicine（华盛顿大学医学院）；University of California, Davis Medical Center（加州大学戴维斯医学中心）
Neural networks promise to bring robust, quantitative analysis to medical fields, but adoption is limited by the technicalities of training these networks. To address this translation gap between medical researchers and neural networks in the field of pathology, we have created an intuitive interface which utilizes the commonly used whole slide image (WSI) viewer, Aperio ImageScope (Leica Biosystems Imaging, Inc.), for the annotation and display of neural network predictions on WSIs. Leveraging this, we propose the use of a human-in-the-loop strategy to reduce the burden of WSI annotation. We …
Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unlabeled, unannotated pathology slides
University of Glasgow（格拉斯哥大学）；New York University School of Medicine（纽约大学医学院）；New York University Langone Health（纽约大学朗格尼医学中心）；Karolinska Institutet（卡罗林斯卡学院）；University College London Hospital（伦敦大学学院医院）
Helmholtz Munich（亥姆霍兹慕尼黑中心）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In many histopathology tasks, sample classification depends on morphological details in tissue or single cells that are only visible at the highest magnification. For a pathologist, this implies tedious zooming in and out, while for a computational decision support algorithm, it leads to the analysis of a huge number of small image patches per whole slide image (WSI). Attention-based multiple instance learning (MIL), where attention estimation is learned in a weakly supervised manner, has been successfully applied in computational histopathology, but it is challenged by large numbers of irrele…
Autonomous labeling of surgical resection margins using a foundation model
University of California, Los Angeles（加州大学洛杉矶分校）；Fatih Sultan Mehmet Vakif University（法提赫·苏丹·穆罕默德·瓦克夫大学）；University of California, Davis（加州大学戴维斯分校）；Hadassah Hebrew University Medical Center（哈达萨希伯来大学医学中心）
University of Freiburg（弗赖堡大学）；Collaborative Research Institute Intelligent Oncology (CRIION)（智能肿瘤学合作研究院（CRIION））；Facultad Latinoamericana de Ciencias Sociales (FLACSO)（拉丁美洲社会科学学院（FLACSO））；National Scientific and Technical Research Council (CONICET)（国家科学技术研究委员会（CONICET））；University Hospital of Munich (LMU)（慕尼黑大学附属医院（LMU））；Medical Center Freiburg（弗赖堡医学中心）
Cancer detection and classification from gigapixel whole slide images of stained tissue specimens has recently experienced enormous progress in computational histopathology. The limitation of available pixel-wise annotated scans shifted the focus from tumor localization to global slide-level classification on the basis of (weakly-supervised) multiple-instance learning despite the clinical importance of local cancer detection. However, the worse performance of these techniques in comparison to fully supervised methods has limited their usage until now for diagnostic interventions in domains of …
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）
Local representation learning has been a key challenge to promote the performance of the histopathological whole slide images analysis. The previous representation learning methods followed the supervised learning paradigm. However, manual annotation for large-scale WSIs is time-consuming and labor-intensive. Hence, the self-supervised contrastive learning has recently attracted intensive attention. The present contrastive learning methods treat each sample as a single class, which suffers from class collision problems, especially in the domain of histopathology image analysis. In this paper, …
Northeastern University（东北大学）；University of the Chinese Academy of Sciences（中国科学院大学）；China Medical University（中国医科大学）
The deep neural network is a research hotspot for histopathological image analysis, which can improve the efficiency and accuracy of diagnosis for pathologists or be used for disease screening. The whole slide pathological image can reach one gigapixel and contains abundant tissue feature information, which needs to be divided into a lot of patches in the training and inference stages. This will lead to a long convergence time and large memory consumption. Furthermore, well-annotated data sets are also in short supply in the field of digital pathology. Inspired by the pathologist's clinical di…
The Chinese University of Hong Kong（香港中文大学）；the Sixth Affiliated Hospital of Sun Yat-sen University（中山大学附属第六医院）；Imsight Medical Technology Co., Ltd.（艾迈医疗科技有限公司）
The whole slide histopathology images (WSIs) play a critical role in gastric cancer diagnosis. However, due to the large scale of WSIs and various sizes of the abnormal area, how to select informative regions and analyze them are quite challenging during the automatic diagnosis process. The multi-instance learning based on the most discriminative instances can be of great benefit for whole slide gastric image diagnosis. In this paper, we design a recalibrated multi-instance deep learning method (RMDL) to address this challenging problem. We first select the discriminative instances, and then u…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）
The rapidly emerging field of computational pathology has the potential to enable objective diagnosis, therapeutic response prediction and identification of new morphological features of clinical relevance. However, deep learning-based computational pathology approaches either require manual annotation of gigapixel whole slide images (WSIs) in fully-supervised settings or thousands of WSIs with slide-level labels in a weakly-supervised setting. Moreover, whole slide level computational pathology methods also suffer from domain adaptation and interpretability issues. These challenges have preve…
Imperial College London（帝国理工学院）；BioMedIA Group, Department of Computing（计算系BioMedIA小组）
Few-shot semantic segmentation (FSS) has great potential for medical imaging applications. Most of the existing FSS techniques require abundant annotated semantic classes for training. However, these methods may not be applicable for medical images due to the lack of annotations. To address this problem we make several contributions: (1) A novel self-supervised FSS framework for medical images in order to eliminate the requirement for annotations during training. Additionally, superpixel-based pseudo-labels are generated to provide supervision; (2) An adaptive local prototype pooling module pl…
Immunohistochemistry guided segmentation of benign epithelial cells, in situ lesions, and invasive epithelial cells in breast cancer slides
Norwegian University of Science and Technology (NTNU)（挪威科技大学）；St. Olavs hospital, Trondheim University Hospital（圣奥拉夫医院，特隆赫姆大学医院）；SINTEF Digital（SINTEF数字研究所）；Sopra Steria（索普瑞斯特利亚）；Uppsala University（乌普萨拉大学）；University of Bergen（卑尔根大学）；Haukeland University Hospital（豪克兰大学医院）
A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images
Department of Computer Science, UiT The Arctic University of Norway（计算机科学系，UiT北极大学）；Department of Oncology, University Hospital of North Norway（肿瘤学系，北挪威大学医院）；Department of Clinical Medicine, UiT The Arctic University of Norway（临床医学系，UiT北极大学）；Department of Community Medicine, UiT The Arctic University of Norway（社区医学系，UiT北极大学）；Department of Medical Biology, UiT The Arctic University of Norway（医学生物学系，UiT北极大学）；Department of Clinical Pathology, University Hospital of North Norway（临床病理学系，北挪威大学医院）
Nuclei segmentation is one of the important tasks for whole slide image analysis in digital pathology. With the drastic advance of deep learning, recent deep networks have demonstrated successful performance of the nuclei segmentation task. However, a major bottleneck to achieving good performance is the cost for annotation. A large network requires a large number of segmentation masks, and this annotation task is given to pathologists, not the public. In this paper, we propose a weakly supervised nuclei segmentation method, which requires only point annotations for training. This method can s…
Central South University（中南大学）；Shaanxi Provincial People’s Hospital（陕西省人民医院）；Xiangya Hospital, Central South University（中南大学湘雅医院）
[Purpose] The pathology is decisive for disease diagnosis, but relies heavily on the experienced pathologists. Recently, pathological artificial intelligence (PAI) is thought to improve diagnostic accuracy and efficiency. However, the high performance of PAI based on deep learning in the laboratory generally cannot be reproduced in the clinic. [Methods] Because the data preparation is important for PAI, the paper has reviewed PAI-related studies in the PubMed database published from January 2017 to February 2022, and 118 studies were included. The in-depth analysis of methods for preparing dat…
DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer
Netherlands Cancer Institute（荷兰癌症研究所）；University of Amsterdam（阿姆斯特丹大学）；Ellogon AI B.V.（Ellogon人工智能有限公司）；Radboud University Medical Center（拉德堡德大学医学中心）
CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学中央高等电力学院）；Gustave Roussy Cancer Campus（古斯塔夫·鲁西癌症校区）；Therapanacea（Therapanacea公司）
In whole slide imaging, commonly used staining techniques based on hematoxylin and eosin (H&E) and immunohistochemistry (IHC) stains accentuate different aspects of the tissue landscape. In the case of detecting metastases, IHC provides a distinct readout that is readily interpretable by pathologists. IHC, however, is a more expensive approach and not available at all medical centers. Virtually generating IHC images from H&E using deep neural networks thus becomes an attractive alternative. Deep generative models such as CycleGANs learn a semantically-consistent mapping between two image domai…
Whole slide images (WSI) are microscopy images of stained tissue slides routinely prepared for diagnosis and treatment selection in medical practice. WSI are very large (gigapixel size) and complex (made of up to millions of cells). The current state-of-the-art (SoTA) approach to classify WSI subdivides them into tiles, encodes them by pre-trained networks and applies Multiple Instance Learning (MIL) to train for specific downstream tasks. However, annotated datasets are often small, typically a few hundred to a few thousand WSI, which may cause overfitting and underperforming models. Converse…
Whole Slide Concepts: A Supervised Foundation Model For Pathological Images
Institute for Digital Medicine MEVIS（数字医学MEVIS研究所）；Fraunhofer（弗劳恩霍夫）；Institute for Toxicology and Experimental Medicine（毒理学与实验医学研究所）；Institute for Diagnostic and Interventional Radiology（诊断与介入放射学研究所）；Hannover Medical School（汉诺威医学院）
UiT The Arctic University of Norway（挪威北极大学）；University Hospital of North Norway（北挪威大学医院）
Increased levels of tumor infiltrating lymphocytes (TILs) in cancer tissue indicate favourable outcomes in many types of cancer. Manual quantification of immune cells is inaccurate and time consuming for pathologists. Our aim is to leverage a computational solution to automatically quantify TILs in whole slide images (WSIs) of standard diagnostic haematoxylin and eosin stained sections (H&E slides) from lung cancer patients. Our approach is to transfer an open source machine learning method for segmentation and classification of nuclei in H&E slides trained on public data to TIL quantification…
Image annotation is one essential prior step to enable data-driven algorithms. In medical imaging, having large and reliably annotated data sets is crucial to recognize various diseases robustly. However, annotator performance varies immensely, thus impacts model training. Therefore, often multiple annotators should be employed, which is however expensive and resource-intensive. Hence, it is desirable that users should annotate unseen data and have an automated system to unobtrusively rate their performance during this process. We examine such a system based on whole slide images (WSIs) showin…
