University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯大学医院及医学院）；National Center for Tumor Diseases (NCT), University Hospital Heidelberg（海德堡大学医院国家肿瘤疾病中心）；Leeds Institute of Medical Research at St James’s, University of Leeds（利兹大学圣詹姆斯利兹医学研究所）
Tim Lenz, Omar S. M. El Nahhas, Marta Ligero, Jakob Nikolas Kather
Deep Learning models have been successfully utilized to extract clinically actionable insights from routinely available histology data. Generally, these models require annotations performed by clinicians, which are scarce and costly to generate. The emergence of self-supervised learning (SSL) methods remove this barrier, allowing for large-scale analyses on non-annotated data. However, recent SSL approaches apply increasingly expansive model architectures and larger datasets, causing the rapid escalation of data volumes, hardware prerequisites, and overall expenses, limiting access to these resources to few institutions. Therefore, we investigated the complexity of contrastive SSL in computational pathology in relation to classification performance with the utilization of consumer-grade hardware. Specifically, we analyzed the effects of adaptations in data volume, architecture, and algorithms on downstream classification tasks, emphasizing their impact on computational resources. We trained breast cancer foundation models on a large public patient cohort and validated them on various downstream classification tasks in a weakly supervised manner on two external public patient cohorts. Our experiments demonstrate that we can improve downstream classification performance whilst reducing SSL training duration by 90%. In summary, we propose a set of adaptations which enable the utilization of SSL in computational pathology in non-resource abundant environments.
Deep learning (DL) has shown great potential in digital pathology applications. The robustness of a diagnostic DL-based solution is essential for safe clinical deployment. In this work we evaluate if adding uncertainty estimates for DL predictions in digital pathology could result in increased value for the clinical applications, by boosting the general predictive performance or by detecting mispredictions. We compare the effectiveness of model-integrated methods (MC dropout and Deep ensembles) with a model-agnostic approach (Test time augmentation, TTA). Moreover, four uncertainty metrics are…
Model-agnostic Knowledge Augmented Machine Learning for Weakly Annotated Bioimaging Data
The University of Melbourne（墨尔本大学）；University of Wollongong（伍伦贡大学）；The University of Sydney（悉尼大学）；The Florey Institute of Neuroscience and Mental Health, The University of Melbourne（墨尔本大学弗洛里神经科学与心理健康研究所）；GenieUs Genomics Pty Ltd（GenieUs Genomics有限公司）；Xstendyl
Sunnybrook Research Institute（桑尼布鲁克研究所）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
To alleviate the burden of gathering detailed expert annotations when training deep neural networks, we propose a weakly supervised learning approach to recognize metastases in microscopic images of breast lymph nodes. We describe an alternative training loss which clusters weakly labeled bags in latent space to inform relevance of patch-instances during training of a convolutional neural network. We evaluate our method on the Camelyon dataset which contains high-resolution digital slides of breast lymph nodes, where labels are provided at the image-level and only subsets of patches are made a…
Regression-based Deep-Learning predicts molecular biomarkers from pathology slides
Else Kroener Fresenius Center for Digital Health（埃尔瑟·克勒纳·弗雷森纽斯数字健康中心）；TUD Dresden University of Technology（德累斯顿工业大学）；University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院及大学医院）；University of Applied Sciences of Western Switzerland (HES-SO Valais)（瑞士西部应用科学大学（瓦莱州））；IBM Research Europe（IBM欧洲研究院）；Centre de Recherche des Cordeliers（科尔得利研究所）；Sorbonne Université（索邦大学）；Université Paris Cité（巴黎城市大学）；Assistance Publique-Hôpitaux de Paris（巴黎公共医疗援助中心）；Université Paris-Est Créteil（巴黎东部克雷泰伊大学）；University Hospital RWTH Aachen（亚琛工业大学附属医院）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；German Cancer Consortium (DKTK)（德国癌症联盟）；University Hospital Heidelberg（海德堡大学医院）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Leeds（利兹大学）
A multimodal vision foundation model for generalizable knee pathology
Beihang University（北京航空航天大学）；Peking University Third Hospital（北京大学第三医院）；Institute of Sports Medicine of Peking University（北京大学运动医学研究所）；Ministry of Education（教育部）
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）
The rapidly emerging field of computational pathology has the potential to enable objective diagnosis, therapeutic response prediction and identification of new morphological features of clinical relevance. However, deep learning-based computational pathology approaches either require manual annotation of gigapixel whole slide images (WSIs) in fully-supervised settings or thousands of WSIs with slide-level labels in a weakly-supervised setting. Moreover, whole slide level computational pathology methods also suffer from domain adaptation and interpretability issues. These challenges have preve…
Bayer AG（拜耳股份公司）；Median Technologies（Median科技公司）；Bayer Crop Science（拜耳作物科学公司）；Pfizer Worldwide Research Development and Medical（辉瑞全球研发与医疗部门）
In pre-clinical pathology, there is a paradox between the abundance of raw data (whole slide images from many organs of many individual animals) and the lack of pixel-level slide annotations done by pathologists. Due to time constraints and requirements from regulatory authorities, diagnoses are instead stored as slide labels. Weakly supervised training is designed to take advantage of those data, and the trained models can be used by pathologists to rank slides by their probability of containing a given lesion of interest. In this work, we propose a novel contextualized eXplainable AI (XAI) f…
Norwegian University of Science and Technology（挪威科技大学）；Innlandet Hospital Trust（内陆医院信托）；University of Oslo（奥斯陆大学）
We propose a novel pathology-sensitive deep learning model (PS-DeVCEM) for frame-level anomaly detection and multi-label classification of different colon diseases in video capsule endoscopy (VCE) data. Our proposed model is capable of coping with the key challenge of colon apparent heterogeneity caused by several types of diseases. Our model is driven by attention-based deep multiple instance learning and is trained end-to-end on weakly labeled data using video labels instead of detailed frame-by-frame annotation. The spatial and temporal features are obtained through ResNet50 and residual Lo…
Joint multi-task learning improves weakly-supervised biomarker prediction in computational pathology
Else Kroener Fresenius Center for Digital Health（Else Kroener Fresenius数字健康中心）；Medical Faculty Carl Gustav Carus（卡尔·古斯塔夫·卡鲁斯医学院）；TUD Dresden University of Technology（德累斯顿工业大学）；School of Computer Science, University of St Andrews（圣安德鲁斯大学计算机学院）；Department of Diagnostic and Interventional Radiology, University Hospital Aachen（亚琛大学医院诊断与介入放射科）；Department of Medicine 1, University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯大学医院第一医学部）；Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg（海德堡大学医院国家肿瘤疾病中心（NCT）医学肿瘤科）
Localization of chest pathologies in chest X-ray images is a challenging task because of their varying sizes and appearances. We propose a novel weakly supervised method to localize chest pathologies using class aware deep multiscale feature learning. Our method leverages intermediate feature maps from CNN layers at different stages of a deep network during the training of a classification model using image level annotations of pathologies. During the training phase, a set of \emph{layer relevance weights} are learned for each pathology class and the CNN is optimized to perform pathology class…
Toward Efficient Weakly Supervised Semantic Segmentation Using Only Low-Magnification Histopathological Images
University of Information Technology, Ho Chi Minh City, Vietnam（越南胡志明市信息科技学院）；Vietnam National University Ho Chi Minh City, Ho Chi Minh City, Vietnam（越南国家大学胡志明市分校）
Brigham and Women’s Hospital（布里格姆妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院布罗德研究所）；Department of Pathology（病理系）；Northwell Health（诺斯韦尔健康）；Department of Biomedical Informatics（生物医学信息学系）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；Cancer Program（癌症项目）；Cancer Data Science（癌症数据科学）
Deep Learning-based computational pathology algorithms have demonstrated profound ability to excel in a wide array of tasks that range from characterization of well known morphological phenotypes to predicting non-human-identifiable features from histology such as molecular alterations. However, the development of robust, adaptable, and accurate deep learning-based models often rely on the collection and time-costly curation large high-quality annotated training data that should ideally come from diverse sources and patient populations to cater for the heterogeneity that exists in such dataset…
Benchmarking Pathology Feature Extractors for Whole Slide Image Classification
University of St Andrews（圣安德鲁斯大学）；Else Kröner Fresenius Center for Digital Health（埃尔瑟·克勒纳·弗雷森纽斯数字健康中心）；TUD Dresden University of Technology（德累斯顿工业大学）；University Hospital Aachen（亚琛大学医院）；University Hospital Heidelberg（海德堡大学医院）；University Hospital Dresden（德累斯顿大学医院）
Towards Segmenting the Invisible: An End-to-End Registration and Segmentation Framework for Weakly Supervised Tumour Analysis
Institute of Technical；Business Information Systems, Faculty of Computer Science, Otto von Guericke University Magdeburg, Magdeburg, Germany Human Technopole, Milan, Italy Institute of Medical Engineering, Faculty of Electrical Engineering；Information Technology, Otto von Guericke University Magdeburg, Magdeburg, Germany Centre for Behavioural Brain Sciences, Magdeburg, Germany
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University Medicine Essen（埃森大学医学院）
When reading images, radiologists generate text reports describing the findings therein. Current state-of-the-art computer-aided diagnosis tools utilize a fixed set of predefined categories automatically extracted from these medical reports for training. This form of supervision limits the potential usage of models as they are unable to pick up on anomalies outside of their predefined set, thus, making it a necessity to retrain the classifier with additional data when faced with novel classes. In contrast, we investigate direct text supervision to break away from this closed set assumption. By…
Evaluating Computational Pathology Foundation Models for Prostate Cancer Grading under Distribution Shifts
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）
Histopathology images; microscopy images of stained tissue biopsies contain fundamental prognostic information that forms the foundation of pathological analysis and diagnostic medicine. However, diagnostics from histopathology images generally rely on a visual cognitive assessment of tissue slides which implies an inherent element of interpretation and hence subjectivity. Access to digitized histopathology images enabled the development of computational systems aiming at reducing manual intervention and automating parts of pathologists' workflow. Specifically, applications of deep learning to…
Cell detection and segmentation is fundamental for all downstream analysis of digital pathology images. However, obtaining the pixel-level ground truth for single cell segmentation is extremely labor intensive. To overcome this challenge, we developed an end-to-end deep learning algorithm to perform both single cell detection and segmentation using only point labels. This is achieved through the combination of different task orientated point label encoding methods and a multi-task scheduler for training. We apply and validate our algorithm on PMS2 stained colon rectal cancer and tonsil tissue …
Weakly supervised semantic segmentation produces pixel-level localization from class labels; however, a classifier trained on such labels is likely to focus on a small discriminative region of the target object. We interpret this phenomenon using the information bottleneck principle: the final layer of a deep neural network, activated by the sigmoid or softmax activation functions, causes an information bottleneck, and as a result, only a subset of the task-relevant information is passed on to the output. We first support this argument through a simulated toy experiment and then propose a meth…
Institut für Medizinische Informatik der Universität zu Lübeck（吕贝克大学医学信息学研究所）；Universität zu Lübeck（吕贝克大学）；INSERM（法国国家健康与医学研究院）
For the weakly supervised task of electrocardiogram (ECG) rhythm classification, convolutional neural networks (CNNs) and long short-term memory (LSTM) networks are two increasingly popular classification models. This work investigates whether a combination of both architectures to so-called convolutional long short-term memory (ConvLSTM) networks can improve classification performances by explicitly capturing morphological as well as temporal features of raw ECG records. In addition, various attention mechanisms are studied to localize and visualize record sections of abnormal morphology and …
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）
Supervised learning tasks such as cancer survival prediction from gigapixel whole slide images (WSIs) are a critical challenge in computational pathology that requires modeling complex features of the tumor microenvironment. These learning tasks are often solved with deep multi-instance learning (MIL) models that do not explicitly capture intratumoral heterogeneity. We develop a novel variance pooling architecture that enables a MIL model to incorporate intratumoral heterogeneity into its predictions. Two interpretability tools based on representative patches are illustrated to probe the biolo…
How similar are two images? In computational pathology, where Whole Slide Images (WSIs) of digitally scanned tissue samples from patients can be multi-gigapixels in size, determination of degree of similarity between two WSIs is a challenging task with a number of practical applications. In this work, we explore a novel strategy based on kernelized Maximum Mean Discrepancy (MMD) analysis for determination of pairwise similarity between WSIs. The proposed approach works by calculating MMD between two WSIs using kernels over deep features of image patches. This allows representation of an entire…
Canon Medical Research Europe（佳能医疗研究欧洲公司）；Heriot-Watt University（赫瑞-瓦特大学）；University of Edinburgh（爱丁堡大学）
Deep learning shows great potential for the domain of digital pathology. An automated digital pathology system could serve as a second reader, perform initial triage in large screening studies, or assist in reporting. However, it is expensive to exhaustively annotate large histology image databases, since medical specialists are a scarce resource. In this paper, we apply the semi-supervised teacher-student knowledge distillation technique proposed by Yalniz et al. (2019) to the task of quantifying prognostic features in colorectal cancer. We obtain accuracy improvements through extending this …
In this work, we introduce a novel framework that employs cluster annotation to boost active learning by reducing the number of human interactions required to train deep neural networks. Instead of annotating single samples individually, humans can also label clusters, producing a higher number of annotated samples with the cost of a small label error. Our experiments show that the proposed framework requires 82% and 87% less human interactions for CIFAR-10 and EuroSAT datasets respectively when compared with the fully-supervised training while maintaining similar performance on the test set.
Pests and diseases are relevant factors for production losses in agriculture and, therefore, promote a huge investment in the prevention and detection of its causative agents. In many countries, Integrated Pest Management is the most widely used process to prevent and mitigate the damages caused by pests and diseases in citrus crops. However, its results are credited by humans who visually inspect the orchards in order to identify the disease symptoms, insects and mite pests. In this context, we design a weakly supervised learning process guided by saliency maps to automatically select regions…
A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification
Bakar Computational Health Sciences Institute, University of California, San Francisco（加州大学旧金山分校巴卡尔计算健康科学研究所）；Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco（加州大学旧金山分校海伦·迪勒家族综合癌症中心）；University of California, Berkeley（加州大学伯克利分校）；Center for Data-driven Insights and Innovation, University of California, Office of the President（加州大学校长办公室数据驱动洞察与创新中心）；Department of Pediatrics, University of California, San Francisco（加州大学旧金山分校儿科系）
Trained using only image class label, deep weakly supervised methods allow image classification and ROI segmentation for interpretability. Despite their success on natural images, they face several challenges over histology data where ROI are visually similar to background making models vulnerable to high pixel-wise false positives. These methods lack mechanisms for modeling explicitly non-discriminative regions which raises false-positive rates. We propose novel regularization terms, which enable the model to seek both non-discriminative and discriminative regions, while discouraging unbalanc…
The paper examines the learning mechanism of adaptive agents over weakly-connected graphs and reveals an interesting behavior on how information flows through such topologies. The results clarify how asymmetries in the exchange of data can mask local information at certain agents and make them totally dependent on other agents. A leader-follower relationship develops with the performance of some agents being fully determined by the performance of other agents that are outside their domain of influence. This scenario can arise, for example, due to intruder attacks by malicious agents or as the …
