Generating crossmodal gene expression from cancer histopathology improves multimodal AI predictions
School of Mathematical & Computational Sciences, Indian Association for the Cultivation of Science（数学与计算科学学院，印度科学培养协会）；The Alan Turing Institute（艾伦·图灵研究所）；Comprehensive Cancer Center, King’s College London（国王学院综合癌症中心）；Department of Computer Science and Engineering, Jadavpur University（计算机科学与工程系，贾瓦德pur大学）；MRC Biostatistics Unit, University of Cambridge（剑桥大学医学研究委员会生物统计学单位）；Department of Biostatistics, Bioinformatics and Biomathematics, Georgetown University（生物统计学、生物信息学与生物数学系，杰斐逊大学）；UCL Cancer Institute, Dept of Medical Physics & Biomedical Engineering, University College London（伦敦大学学院癌症研究所，医学物理与生物医学工程系）
Samiran Dey, Christopher R. S. Banerji, Partha Basuchowdhuri, Sanjoy K. Saha, Deepak Parashar, Tapabrata Chakraborti
Emerging research has highlighted that artificial intelligence-based multimodal fusion of digital pathology and transcriptomic features can improve cancer diagnosis (grading/subtyping) and prognosis (survival risk) prediction. However, such direct fusion is impractical in clinical settings, where histopathology remains the gold standard and transcriptomic tests are rarely requested in public healthcare. We experiment on two publicly available multimodal datasets, The Cancer Genomic Atlas and the Clinical Proteomic Tumor Analysis Consortium, spanning four independent cohorts: glioma-glioblastoma, renal, uterine, and breast, and observe significant performance gains in gradation and risk estimation (p-value<0.05) when incorporating synthesized transcriptomic data with WSIs. Also, predictions using synthesized features were statistically close to those obtained with real transcriptomic data (p-value>0.05), consistently across cohorts. Here we show that with our diffusion based crossmodal generative AI model, PathGen, gene expressions synthesized from digital histopathology jointly predict cancer grading and patient survival risk with high accuracy (state-of-the-art performance), certainty (through conformal coverage guarantee) and interpretability (through distributed co-attention maps). PathGen code is available for open use on GitHub at https://github.com/Samiran-Dey/PathGen.
CFCML: A Coarse-to-Fine Crossmodal Learning Framework For Disease Diagnosis Using Multimodal Images and Tabular Data
College of Intelligence and Computing（智能与计算学院）；Tianjin University（天津大学）；Medical School of Tianjin University（天津大学医学院）；Peng Cheng Lab（鹏城实验室）；Department of Statistics and Actuarial Science, School of Computing and Data Science, The University of Hong Kong（统计与精算系，计算与数据科学学院，香港大学）；Department of Radiology, Tianjin Huanhu Hospital（天津华医院放射科）；Tianjin Key Laboratory of Cerebral Vascular and Neurodegenerative Diseases（天津脑血管与神经退行性疾病重点实验室）
University of Houston（休斯顿大学）；University Hospital of Cologne（科隆大学医院）；Weill Cornell Medicine（威尔康奈尔医学院）；Cornell University（康奈尔大学）；University of Turin（都灵大学）
Self-supervised learning (SSL) methods are enabling an increasing number of deep learning models to be trained on image datasets in domains where labels are difficult to obtain. These methods, however, struggle to scale to the high resolution of medical imaging datasets, where they are critical for achieving good generalization on label-scarce medical image datasets. In this work, we propose the Histopathology DatasetGAN (HDGAN) framework, an extension of the DatasetGAN semi-supervised framework for image generation and segmentation that scales well to large-resolution histopathology images. W…
Histopathology image embedding is an active research area in computer vision. Most of the embedding models exclusively concentrate on a specific magnification level. However, a useful task in histopathology embedding is to train an embedding space regardless of the magnification level. Two main approaches for tackling this goal are domain adaptation and domain generalization, where the target magnification levels may or may not be introduced to the model in training, respectively. Although magnification adaptation is a well-studied topic in the literature, this paper, to the best of our knowle…
NYU Grossman School of Medicine（纽约大学格罗斯曼医学院）；University of Toronto（多伦多大学）；Microsoft Research（微软研究院）
Scarcity of labeled histopathology data limits the applicability of deep learning methods to under-profiled cancer types and labels. Transfer learning allows researchers to overcome the limitations of small datasets by pre-training machine learning models on larger datasets similar to the small target dataset. However, similarity between datasets is often determined heuristically. In this paper, we propose a principled notion of distance between histopathology datasets based on a hierarchical generalization of optimal transport distances. Our method does not require any training, is agnostic t…
While microscopic analysis of histopathological slides is generally considered as the gold standard method for performing cancer diagnosis and grading, the current method for analysis is extremely time consuming and labour intensive as it requires pathologists to visually inspect tissue samples in a detailed fashion for the presence of cancer. As such, there has been significant recent interest in computer aided diagnosis systems for analysing histopathological slides for cancer grading to aid pathologists to perform cancer diagnosis and grading in a more efficient, accurate, and consistent ma…
Histopathology Image Normalization via Latent Manifold Compaction
Knight Cancer Institute, Oregon Health and Science University（骑士癌症研究所，俄勒冈健康与科学大学）；Brenden-Colson Center for Pancreatic Care, Oregon Health and Science University（布雷登-科尔森胰腺护理中心，俄勒冈健康与科学大学）
Shiraz University of Technology（设拉子理工大学）；National Brain Mapping Laboratory（国家脑图谱实验室）
We propose a two-step domain shift-invariant mitosis cell detection method based on Faster RCNN and a convolutional neural network (CNN). We generate various domain-shifted versions of existing histopathology images using a stain augmentation technique, enabling our method to effectively learn various stain domains and achieve better generalization. The performance of our method is evaluated on the preliminary test data set of the MIDOG-2021 challenge. The experimental results demonstrate that the proposed mitosis detection method can achieve promising performance for domain-shifted histopatho…
Histopathology images are gigapixel-sized and include features and information at different resolutions. Collecting annotations in histopathology requires highly specialized pathologists, making it expensive and time-consuming. Self-training can alleviate annotation constraints by learning from both labeled and unlabeled data, reducing the amount of annotations required from pathologists. We study the design of teacher-student self-training systems for Non-alcoholic Steatohepatitis (NASH) using clinical histopathology datasets with limited annotations. We evaluate the models on in-distribution…
Histopathology image analysis can be considered as a Multiple instance learning (MIL) problem, where the whole slide histopathology image (WSI) is regarded as a bag of instances (i.e, patches) and the task is to predict a single class label to the WSI. However, in many real-life applications such as computational pathology, discovering the key instances that trigger the bag label is of great interest because it provides reasons for the decision made by the system. In this paper, we propose a deep convolutional neural network (CNN) model that addresses the primary task of a bag classification o…
Prompt-Free Lightweight SAM Adaptation for Histopathology Nuclei Segmentation with Strong Cross-Dataset Generalization
School of Information and Communication Technology, Griffith University, QLD, Australia（信息与通信技术学院，格里菲斯大学，昆士兰，澳大利亚）；School of MDP -- Clinical Medicine, Griffith University, QLD, Australia（MDP临床医学学院，格里菲斯大学，昆士兰，澳大利亚）
Stony Brook Medicine（石溪大学医学中心）；Stony Brook University（石溪大学）；Cold Spring Harbor Laboratory（冷泉港实验室）
Histopathology remains the gold standard for diagnosis of various cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for various tasks, including immune cell detection and microsatellite instability classification. The state-of-the-art for each task often employs base architectures that have been pretrained for image classification on ImageNet. The standard approach to develop classifiers in histopathology tends to focus narrowly on optimizing models for a single task, not considering the aspects of modeling innovatio…
Spatial Transcriptomics Expression Prediction from Histopathology Based on Cross-Modal Mask Reconstruction and Contrastive Learning
University of Regensburg（雷根斯堡大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院）；Hanover Medical School（汉诺威医学院）；University Clinic Freiburg（弗莱堡大学附属医院）；Fraunhofer Institute for Digital Medicine MEVIS（弗劳恩霍夫数字医学研究所）
University of British Columbia（不列颠哥伦比亚大学）；Eindhoven University of Technology（埃因霍温理工大学）；Massachusetts General Hospital（马萨诸塞州综合医院）；Harvard Medical School（哈佛医学院）；BC Cancer Agency（不列颠哥伦比亚癌症机构）
Visual microscopic study of diseased tissue by pathologists has been the cornerstone for cancer diagnosis and prognostication for more than a century. Recently, deep learning methods have made significant advances in the analysis and classification of tissue images. However, there has been limited work on the utility of such models in generating histopathology images. These synthetic images have several applications in pathology including utilities in education, proficiency testing, privacy, and data sharing. Recently, diffusion probabilistic models were introduced to generate high quality ima…
Preparing and scanning histopathology slides consists of several steps, each with a multitude of parameters. The parameters can vary between pathology labs and within the same lab over time, resulting in significant variability of the tissue appearance that hampers the generalization of automatic image analysis methods. Typically, this is addressed with ad-hoc approaches such as staining normalization that aim to reduce the appearance variability. In this paper, we propose a systematic solution based on domain-adversarial neural networks. We hypothesize that removing the domain information fro…
xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Machine Learning Group, Technische Universität Berlin（柏林工业大学机器学习组）；Aignostics GmbH（艾格诺斯蒂克斯有限公司）；Institute of Pathology, Ludwig Maximilian University（路德维希-马克西米利安大学病理学研究所）；German Cancer Research Center（德国癌症研究中心）；Institute of Pathology, Charité Universitätsmedizin（夏里特医学院病理学研究所）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）；Max-Planck Institute for Informatics（马克斯·普朗克信息学研究所）
We propose a virtual staining methodology based on Generative Adversarial Networks to map histopathology images of breast cancer tissue from H&E stain to PHH3 and vice versa. We use the resulting synthetic images to build Convolutional Neural Networks (CNN) for automatic detection of mitotic figures, a strong prognostic biomarker used in routine breast cancer diagnosis and grading. We propose several scenarios, in which CNN trained with synthetically generated histopathology images perform on par with or even better than the same baseline model trained with real images. We discuss the potentia…
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）；Vector Institute（矢量研究院）
Recent advances in digital pathology have led to the need for Histopathology Image Retrieval (HIR) systems that search through databases of biopsy images to find similar cases to a given query image. These HIR systems allow pathologists to effortlessly and efficiently access thousands of previously diagnosed cases in order to exploit the knowledge in the corresponding pathology reports. Since HIR systems may have to deal with millions of gigapixel images, the extraction of compact and expressive image features must be available to allow for efficient and accurate retrieval. In this paper, we p…
CLEAR: Causal Learning Framework For Robust Histopathology Tumor Detection Under Out-Of-Distribution Shifts
VNU University of Engineering and Technology（越南国立大学工程与技术大学）；College of Engineering and Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）；VinUni-Illinois Smart Health Center, VinUniversity（VinUni-Illinois智能健康中心，VinUniversity）
Dartmouth College（达特茅斯学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希区柯克医疗中心）
We present an image translation approach to generate augmented data for mitigating data imbalances in a dataset of histopathology images of colorectal polyps, adenomatous tumors that can lead to colorectal cancer if left untreated. By applying cycle-consistent generative adversarial networks (CycleGANs) to a source domain of normal colonic mucosa images, we generate synthetic colorectal polyp images that belong to diagnostically less common polyp classes. Generated images maintain the general structure of their source image but exhibit adenomatous features that can be enhanced with our propose…
University of Waterloo（滑铁卢大学）；Amirkabir University of Technology（阿米尔卡比尔理工大学）
We investigate the concept of deep barcodes and propose two methods to generate them in order to expedite the process of classification and retrieval of histopathology images. Since binary search is computationally less expensive, in terms of both speed and storage, deep barcodes could be useful when dealing with big data retrieval. Our experiments use the dataset Kimia Path24 to test three pre-trained networks for image retrieval. The dataset consists of 27,055 training images in 24 different classes with large variability, and 1,325 test images for testing. Apart from the high-speed and effi…
Histopathology images; microscopy images of stained tissue biopsies contain fundamental prognostic information that forms the foundation of pathological analysis and diagnostic medicine. However, diagnostics from histopathology images generally rely on a visual cognitive assessment of tissue slides which implies an inherent element of interpretation and hence subjectivity. Access to digitized histopathology images enabled the development of computational systems aiming at reducing manual intervention and automating parts of pathologists' workflow. Specifically, applications of deep learning to…
Nuclei segmentation is a fundamental task that is critical for various computational pathology applications including nuclei morphology analysis, cell type classification, and cancer grading. Conventional vision-based methods for nuclei segmentation struggle in challenging cases and deep learning approaches have proven to be more robust and generalizable. However, CNNs require large amounts of labeled histopathology data. Moreover, conventional CNN-based approaches lack structured prediction capabilities which are required to distinguish overlapping and clumped nuclei. Here, we present an appr…
Advancement in digital pathology and artificial intelligence has enabled deep learning-based computer vision techniques for automated disease diagnosis and prognosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized, making them infeasible to be used directly for training deep neural networks. Hence, for modeling, a two-stage approach is adopted: Patch representations are extracted first, followed by the aggregation for WSI prediction. These approaches require detailed pixel-level annotations for training the patch encoder. However, obtaining thes…
Are nuclear masks all you need for improved out-of-domain generalisation? A closer look at cancer classification in histopathology
University of Oslo（奥斯陆大学）；Otto-von-Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）；Singapore Institute of Technology（新加坡科技学院）；Oslo University Hospital（奥斯陆大学医院）；UiT The Arctic University of Norway（挪威北极大学）
Dartmouth College（达特茅斯学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希区柯克医疗中心）
The unique nature of histopathology images opens the door to domain-specific formulations of image translation models. We propose a difficulty translation model that modifies colorectal histopathology images to be more challenging to classify. Our model comprises a scorer, which provides an output confidence to measure the difficulty of images, and an image translator, which learns to translate images from easy-to-classify to hard-to-classify using a training set defined by the scorer. We present three findings. First, generated images were indeed harder to classify for both human pathologists…
Retrieval-Guided Generation for Safer Histopathology Image Captioning
Kimia Lab, Department of Artificial Intelligence \& Informatics, Mayo Clinic, Rochester, MN, USA；Department of Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock, AR, USA；Lane Department of Computer Science；Electrical Engineering, West Virginia University, Morgantown, WV, USA；Department of Computer Science；Engineering, Princeton University, Princeton, NJ, USA；Department of Computer Sciences, University of Wisconsin--Madison, Madison, WI, USA
LDCSF: Local depth convolution-based Swim framework for classifying multi-label histopathology images
College of Computer Science and Electronic Engineering（计算机科学与电子工程学院）；Third Xiangya Hospital（湘雅三医院）；Hunan University（湖南大学）；Central South University（中南大学）
The Pennsylvania State University（宾夕法尼亚州立大学）；Lister Hill National Center for Biomedical Communications（利斯特希尔国家生物医学通信中心）；National Library of Medicine（国家医学图书馆）；Penn State Health Milton S. Hershey Medical Center（宾夕法尼亚州立大学健康米尔顿·S·赫尔希医学中心）；Penn State College Of Medicine（宾夕法尼亚州立大学医学院）
Histopathological analysis is the present gold standard for precancerous lesion diagnosis. The goal of automated histopathological classification from digital images requires supervised training, which requires a large number of expert annotations that can be expensive and time-consuming to collect. Meanwhile, accurate classification of image patches cropped from whole-slide images is essential for standard sliding window based histopathology slide classification methods. To mitigate these issues, we propose a carefully designed conditional GAN model, namely HistoGAN, for synthesizing realisti…
Equivariant Imaging Biomarkers for Robust Unsupervised Segmentation of Histopathology
Department of Biomedical Engineering, Yale University（生物医学工程系，耶鲁大学）；Department of Radiology & Biomedical Imaging, Yale University（放射科与生物医学成像系，耶鲁大学）；Department of Urology, Yale University（泌尿外科系，耶鲁大学）；Medical Scientist Training Program, Yale University（医学科学家培训计划，耶鲁大学）
Controllable Diffusion-Based Lesion Inpainting for Scalable Histopathology Data Augmentation
Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong（香港大学李嘉诚医学院诊断放射学系）；Department of Pathology, Li Ka Shing Faculty of Medicine, The University of Hong Kong（香港大学李嘉诚医学院病理学系）；Department of Anatomical and Cellular Pathology, Prince of Wales Hospital, The Chinese University of Hong Kong（香港中文大学威尔斯亲王医院解剖及细胞病理学系）；Department of Infectious Diseases and Public Health, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong（香港城市大学赛马会动物医学及生命科学院传染病及公共卫生学系）；CityU Veterinary Diagnostic Laboratory Co., Ltd., City University of Hong Kong（香港城市大学城市大学兽医诊断实验室有限公司）
It is commonly recognized that color variations caused by differences in stains is a critical issue for histopathology image analysis. Existing methods adopt color matching, stain separation, stain transfer or the combination of them to alleviate the stain variation problem. In this paper, we propose a novel Stain-Adaptive Self-Supervised Learning(SASSL) method for histopathology image analysis. Our SASSL integrates a domain-adversarial training module into the SSL framework to learn distinctive features that are robust to both various transformations and stain variations. The proposed SASSL i…
