The Segment Anything Model (SAM) marks a significant advancement in segmentation models, offering robust zero-shot abilities and dynamic prompting. However, existing medical SAMs are not suitable for the multi-scale nature of whole-slide images (WSIs), restricting their effectiveness. To resolve this drawback, we present WSI-SAM, enhancing SAM with precise object segmentation capabilities for histopathology images using multi-resolution patches, while preserving its efficient, prompt-driven design, and zero-shot abilities. To fully exploit pretrained knowledge while minimizing training overhead, we keep SAM frozen, introducing only minimal extra parameters and computational overhead. In particular, we introduce High-Resolution (HR) token, Low-Resolution (LR) token and dual mask decoder. This decoder integrates the original SAM mask decoder with a lightweight fusion module that integrates features at multiple scales. Instead of predicting a mask independently, we integrate HR and LR token at intermediate layer to jointly learn features of the same object across multiple resolutions. Experiments show that our WSI-SAM outperforms state-of-the-art SAM and its variants. In particular, our model outperforms SAM by 4.1 and 2.5 percent points on a ductal carcinoma in situ (DCIS) segmentation tasks and breast cancer metastasis segmentation task (CAMELYON16 dataset). The code will be available at https://github.com/HongLiuuuuu/WSI-SAM.
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）
Transformer has been widely used in histopathology whole slide image (WSI) classification for the purpose of tumor grading, prognosis analysis, etc. However, the design of token-wise self-attention and positional embedding strategy in the common Transformer limits the effectiveness and efficiency in the application to gigapixel histopathology images. In this paper, we propose a kernel attention Transformer (KAT) for histopathology WSI classification. The information transmission of the tokens is achieved by cross-attention between the tokens and a set of kernels related to a set of positional …
University of Rochester（罗切斯特大学）；HistoWiz Inc.（HistoWiz公司）
Automated whole slide image (WSI) tagging has become a growing demand due to the increasing volume and diversity of WSIs collected nowadays in histopathology. Various methods have been studied to classify WSIs with single tags but none of them focuses on labeling WSIs with multiple tags. To this end, we propose a novel end-to-end trainable deep neural network named Patch Transformer which can effectively predict multiple slide-level tags from WSI patches based on both the correlations and the uniqueness between the tags. Specifically, the proposed method learns patch characteristics considerin…
Pan-cancer Histopathology WSI Pre-training with Position-aware Masked Autoencoder
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）；University of Science and Technology of China（中国科学技术大学）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程高精尖创新中心）；School of Astronautics, Beihang University（北京航空航天大学航天学院）；School of Engineering Medicine, Beihang University（北京航空航天大学工程医学院）；School of Software, Hefei University of Technology（合肥工业大学软件学院）；the First Affiliated Hospital of USTC（中国科学技术大学第一附属医院）
Promptable Representation Distribution Learning and Data Augmentation for Gigapixel Histopathology WSI Analysis
Beihang University（北京航空航天大学）；Tianmushan Laboratory（天目山实验室）；Hefei University of Technology（合肥工业大学）；the First Affiliated Hospital of USTC（中国科学技术大学附属第一医院）；University of Science and Technology of China（中国科学技术大学）
Extracting rich phenotype information, such as cell density and arrangement, from whole slide histology images (WSIs), requires analysis of large field of view, i.e more contexual information. This can be achieved through analyzing the digital slides at lower resolution. A potential drawback is missing out on details present at a higher resolution. To jointly leverage complementary information from multiple resolutions, we present a novel transformer based Pyramidal Context-Detail Network (CD-Net). CD-Net exploits the WSI pyramidal structure through co-training of proposed Context and Detail M…
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）
Local representation learning has been a key challenge to promote the performance of the histopathological whole slide images analysis. The previous representation learning methods followed the supervised learning paradigm. However, manual annotation for large-scale WSIs is time-consuming and labor-intensive. Hence, the self-supervised contrastive learning has recently attracted intensive attention. The present contrastive learning methods treat each sample as a single class, which suffers from class collision problems, especially in the domain of histopathology image analysis. In this paper, …
Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis
University of Science and Technology of China（中国科学技术大学）；the First Affiliated Hospital of USTC（中国科学技术大学第一附属医院）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程高精尖创新中心）
Automated classification of histopathological whole-slide images (WSI) of breast tissue requires analysis at very high resolutions with a large contextual area. In this paper, we present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, ductal carcinoma in situ (DCIS), and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution patches to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stac…
Advancement in digital pathology and artificial intelligence has enabled deep learning-based computer vision techniques for automated disease diagnosis and prognosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized, making them infeasible to be used directly for training deep neural networks. Hence, for modeling, a two-stage approach is adopted: Patch representations are extracted first, followed by the aggregation for WSI prediction. These approaches require detailed pixel-level annotations for training the patch encoder. However, obtaining thes…
Central South University（中南大学）；Nanjing Tech University（南京工业大学）；The third affiliated hospital of Soochow University（苏州大学第三附属医院）
Digital pathological analysis is run as the main examination used for cancer diagnosis. Recently, deep learning-driven feature extraction from pathology images is able to detect genetic variations and tumor environment, but few studies focus on differential gene expression in tumor cells. In this paper, we propose a self-supervised contrastive learning framework, HistCode, to infer differential gene expressions from whole slide images (WSIs). We leveraged contrastive learning on large-scale unannotated WSIs to derive slide-level histopathological feature in latent space, and then transfer it t…
Semantic-Aware Subgraph State Space Model for WSI Classification in Histopathology
National College for Excellent Engineers（卓越工程师学院）；Beihang University（北京航空航天大学）；School of Biological Science and Medical Engineering（生物与医学工程学院）；College of Engineering（工学院）；Peking University（北京大学）
From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformer
University of Science and Technology Beijing（北京科技大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；Changhai Hospital（长海医院）；Peking University（北京大学）
In the clinical setting of histopathology, whole-slide image (WSI) artifacts frequently arise, distorting regions of interest, and having a pernicious impact on WSI analysis. Image-to-image translation networks such as CycleGANs are in principle capable of learning an artifact removal function from unpaired data. However, we identify a surjection problem with artifact removal, and propose an weakly-supervised extension to CycleGAN to address this. We assemble a pan-cancer dataset comprising artifact and clean tiles from the TCGA database. Promising results highlight the soundness of our method…
Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University（北京航空航天大学生物医学工程北京高级创新中心）；School of Software, Hefei University of Technology（合肥工业大学软件学院）；Tianjin Institute of Industrial Biotechnology, Chinese Academy of Sciences（中国科学院天津工业生物技术研究所）
Content-based histopathological image retrieval (CBHIR) has become popular in recent years in the domain of histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. While, it is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database that consists of histopathological whole slide images (WSIs) for a query ROI. In this paper, we propose a novel framework for …
Helmholtz Munich（亥姆霍兹慕尼黑中心）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In many histopathology tasks, sample classification depends on morphological details in tissue or single cells that are only visible at the highest magnification. For a pathologist, this implies tedious zooming in and out, while for a computational decision support algorithm, it leads to the analysis of a huge number of small image patches per whole slide image (WSI). Attention-based multiple instance learning (MIL), where attention estimation is learned in a weakly supervised manner, has been successfully applied in computational histopathology, but it is challenged by large numbers of irrele…
MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction using Integrative Histopathology-Genomic Features
Heilongjiang University（黑龙江大学）；La Trobe University（拉筹伯大学）；University of South China（南华大学）；The First Affiliated Hospital（第一附属医院）；Hengyang Medical School（衡阳医学院）；School of Basic Medical Sciences（基础医学院）
Hierarchical Classification for Improved Histopathology Image Analysis
School of Electrical Engineering, Korea University, Seoul, Korea（韩国大学电气工程学院）；Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul, Korea（韩国天主大学医学院医院病理系）
Histopathology image analysis can be considered as a Multiple instance learning (MIL) problem, where the whole slide histopathology image (WSI) is regarded as a bag of instances (i.e, patches) and the task is to predict a single class label to the WSI. However, in many real-life applications such as computational pathology, discovering the key instances that trigger the bag label is of great interest because it provides reasons for the decision made by the system. In this paper, we propose a deep convolutional neural network (CNN) model that addresses the primary task of a bag classification o…
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
Representation learning for Whole Slide Images (WSIs) is pivotal in developing image-based systems to achieve higher precision in diagnostic pathology. We propose a two-stage framework for WSI representation learning. We sample relevant patches using a color-based method and use graph neural networks to learn relations among sampled patches to aggregate the image information into a single vector representation. We introduce attention via graph pooling to automatically infer patches with higher relevance. We demonstrate the performance of our approach for discriminating two sub-types of lung ca…
University of Waterloo（滑铁卢大学）；Huron Digital Pathology（休伦数字病理学公司）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）；Vector Institute（矢量研究所）
With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform milli…
Weakly Supervised Contrastive Learning for Histopathology Patch Embeddings
Department of Electrical and Computer Engineering, University of Utah（电气与计算机工程系，犹他大学）；Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
Benchmarking Histopathology Foundation Models for Ovarian Cancer Bevacizumab Treatment Response Prediction from Whole Slide Images
Faculty of Science, University of British Columbia（不列颠哥伦比亚大学理学院）；School of Biomedical Engineering, University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）；Department of Pathology and Laboratory Medicine, University of British Columbia（不列颠哥伦比亚大学病理学与实验医学系）；Canada’s Michael Smith Genome Sciences Centre, BC Cancer Research Institute（加拿大迈克尔·史密斯基因组科学中心，BC癌症研究所）
Learning suitable Whole slide images (WSIs) representations for efficient retrieval systems is a non-trivial task. The WSI embeddings obtained from current methods are in Euclidean space not ideal for efficient WSI retrieval. Furthermore, most of the current methods require high GPU memory due to the simultaneous processing of multiple sets of patches. To address these challenges, we propose a novel framework for learning binary and sparse WSI representations utilizing a deep generative modelling and the Fisher Vector. We introduce new loss functions for learning sparse and binary permutation-…
Joint analysis of multiple biomarker images and tissue morphology is important for disease diagnosis, treatment planning and drug development. It requires cross-staining comparison among Whole Slide Images (WSIs) of immuno-histochemical and hematoxylin and eosin (H&E) microscopic slides. However, automatic, and fast cross-staining alignment of enormous gigapixel WSIs at single-cell precision is challenging. In addition to morphological deformations introduced during slide preparation, there are large variations in cell appearance and tissue morphology across different staining. In this paper, …
University of Liverpool（利物浦大学）；Cixi Institute of Biomedical Engineering, Chinese Academy of Sciences（中国科学院慈溪生物医学工程研究所）；China Science IntelliCloud Technology Co., Ltd（中科智云科技有限公司）；Remark AI UK Limited（Remark AI英国有限公司）
Multiple instance learning (MIL) has been increasingly used in the classification of histopathology whole slide images (WSIs). However, MIL approaches for this specific classification problem still face unique challenges, particularly those related to small sample cohorts. In these, there are limited number of WSI slides (bags), while the resolution of a single WSI is huge, which leads to a large number of patches (instances) cropped from this slide. To address this issue, we propose to virtually enlarge the number of bags by introducing the concept of pseudo-bags, on which a double-tier MIL f…
Clinically-Informed Modeling for Pediatric Brain Tumor Classification from Whole-Slide Histopathology Images
Dept. of Computer Science（计算机科学系）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；School of Information（信息学院）；Dell Children's Medical Center（德尔儿童医学中心）；Dept. of OREI（OREI部门）；University of Nevada, Reno（内华达大学里诺分校）
M4: Multi-Proxy Multi-Gate Mixture of Experts Network for Multiple Instance Learning in Histopathology Image Analysis
Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences（中国科学院杭州医学研究所）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）
One Model to Magnify Them All: Efficient Scale-Invariant Histopathology via Conditional Normalization and Continuous Magnification Training
Sano Centre for Computational Medicine（萨诺计算医学中心）；AGH University of Krakow（克拉科夫AGH大学）；Institute of Informatics, HES-SO Valais-Wallis（瓦莱州瓦利斯高等专业学院信息学研究所）；Faculty of Medicine, University of Geneva (UNIGE)（日内瓦大学医学院）
National Institute of Standards and Technology（国家标准与技术研究院）；Rice University（里士满大学）；Department of Physics, Colorado State University（科罗拉多州立大学物理系）；Electrical Engineering Department, University of Colorado（科罗拉多大学电气工程系）
Vector Institute（向量研究所）；Pierre and Marie Curie University（皮埃尔和玛丽·居里大学）；Sorbonne University（索邦大学）；University of Waterloo（滑铁卢大学）
Whole Slide Imaging (WSI) has become an important topic during the last decade. Even though significant progress in both medical image processing and computational resources has been achieved, there are still problems in WSI that need to be solved. A major challenge is the scan size. The dimensions of digitized tissue samples may exceed 100,000 by 100,000 pixels causing memory and efficiency obstacles for real-time processing. The main contribution of this work is representing a WSI by selecting a small number of patches for algorithmic processing (e.g., indexing and search). As a result, we r…
Faculty of Informatics and Data Science, University of Regensburg, Germany（信息学与数据科学学院，雷根萨大学）；Indian Institute of Information Technology Bhopal, India（比哈尔信息学院）
Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading
Dartmouth College（达特茅斯学院）；Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Massachusetts General Hospital（麻省总医院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希契科克医疗中心）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathology
University of Tübingen（蒂宾根大学）；University of Lucerne（卢塞恩大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Institut du Cancer de l’Ouest（西部癌症研究所）；Oncode Institute（Oncode研究所）
The Chinese University of Hong Kong（香港中文大学）；the Sixth Affiliated Hospital of Sun Yat-sen University（中山大学附属第六医院）；Imsight Medical Technology Co., Ltd.（艾迈医疗科技有限公司）
The whole slide histopathology images (WSIs) play a critical role in gastric cancer diagnosis. However, due to the large scale of WSIs and various sizes of the abnormal area, how to select informative regions and analyze them are quite challenging during the automatic diagnosis process. The multi-instance learning based on the most discriminative instances can be of great benefit for whole slide gastric image diagnosis. In this paper, we design a recalibrated multi-instance deep learning method (RMDL) to address this challenging problem. We first select the discriminative instances, and then u…
PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）；David Geffen School of Medicine, UCLA（加州大学洛杉矶分校大卫·格芬医学院）；Allen Institute for AI (AI2)（艾伦人工智能研究所）
