Recent advancements in foundation models have transformed computer vision, driving significant performance improvements across diverse domains, including digital histopathology. However, the advantages of domain-specific histopathology foundation models over general-purpose models for specialized tasks such as cell analysis remain underexplored. This study investigates the representation learning gap between these two categories by analyzing multi-level patch embeddings applied to cell instance segmentation and classification. We implement an encoder-decoder architecture with a consistent decoder and various encoders. These include convolutional, vision transformer (ViT), and hybrid encoders pre-trained on ImageNet-22K or LVD-142M, representing general-purpose foundation models. These are compared against ViT encoders from the recently released UNI, Virchow2, and Prov-GigaPath foundation models, trained on patches extracted from hundreds of thousands of histopathology whole-slide images. The decoder integrates patch embeddings from different encoder depths via skip connections to generate semantic and distance maps. These maps are then post-processed to create instance segmentation masks where each label corresponds to an individual cell and to perform cell-type classification. All encoders remain frozen during training to assess their pre-trained feature extraction capabilities. Using the PanNuke and CoNIC histopathology datasets, and the newly introduced Nissl-stained CytoDArk0 dataset for brain cytoarchitecture studies, we evaluate instance-level detection, segmentation accuracy, and cell-type classification. This study provides insights into the comparative strengths and limitations of general-purpose vs. histopathology foundation models, offering guidance for model selection in cell-focused histopathology and brain cytoarchitecture analysis workflows.
University of Rochester（罗切斯特大学）；HistoWiz Inc.（HistoWiz公司）
Automated whole slide image (WSI) tagging has become a growing demand due to the increasing volume and diversity of WSIs collected nowadays in histopathology. Various methods have been studied to classify WSIs with single tags but none of them focuses on labeling WSIs with multiple tags. To this end, we propose a novel end-to-end trainable deep neural network named Patch Transformer which can effectively predict multiple slide-level tags from WSI patches based on both the correlations and the uniqueness between the tags. Specifically, the proposed method learns patch characteristics considerin…
Learning discriminative representations of unlabelled data is a challenging task. Contrastive self-supervised learning provides a framework to learn meaningful representations using learned notions of similarity measures from simple pretext tasks. In this work, we propose a simple and efficient framework for self-supervised image segmentation using contrastive learning on image patches, without using explicit pretext tasks or any further labeled fine-tuning. A fully convolutional neural network (FCNN) is trained in a self-supervised manner to discern features in the input images and obtain con…
We analyze the effect of offline and online triplet mining for colorectal cancer (CRC) histopathology dataset containing 100,000 patches. We consider the extreme, i.e., farthest and nearest patches to a given anchor, both in online and offline mining. While many works focus solely on selecting the triplets online (batch-wise), we also study the effect of extreme distances and neighbor patches before training in an offline fashion. We analyze extreme cases' impacts in terms of embedding distance for offline versus online mining, including easy positive, batch semi-hard, batch hard triplet minin…
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）
Digital pathology is one of the most significant developments in modern medicine. Pathological examinations are the gold standard of medical protocols and play a fundamental role in diagnosis. Recently, with the advent of digital scanners, tissue histopathology slides can now be digitized and stored as digital images. As a result, digitized histopathological tissues can be used in computer-aided image analysis programs and machine learning techniques. Detection and segmentation of nuclei are some of the essential steps in the diagnosis of cancers. Recently, deep learning has been used for nucl…
Weakly Supervised Contrastive Learning for Histopathology Patch Embeddings
Department of Electrical and Computer Engineering, University of Utah（电气与计算机工程系，犹他大学）；Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
Representation learning for Whole Slide Images (WSIs) is pivotal in developing image-based systems to achieve higher precision in diagnostic pathology. We propose a two-stage framework for WSI representation learning. We sample relevant patches using a color-based method and use graph neural networks to learn relations among sampled patches to aggregate the image information into a single vector representation. We introduce attention via graph pooling to automatically infer patches with higher relevance. We demonstrate the performance of our approach for discriminating two sub-types of lung ca…
Learning from the Right Patches: A Two-Stage Wavelet-Driven Masked Autoencoder for Histopathology Representation Learning
PLRI Medical Informatics Institute（PLRI医学信息学研究所）；CAIMed Reserch Center（CAIMed研究中心）；Hannover Medical School（汉诺威医学院）；Computer Science Institute（计算机科学研究所）；Leibniz University Hannover（莱比锡大学汉诺威分校）；Sanford Burnham Prebys Medical Discovery Institute（桑福医疗发现研究所）；Rady Children’s Institute for Genomic Medicine（拉迪儿童基因医学研究所）；University of California San Diego（加州大学圣地亚哥分校）
Vector Institute（向量研究所）；Pierre and Marie Curie University（皮埃尔和玛丽·居里大学）；Sorbonne University（索邦大学）；University of Waterloo（滑铁卢大学）
Whole Slide Imaging (WSI) has become an important topic during the last decade. Even though significant progress in both medical image processing and computational resources has been achieved, there are still problems in WSI that need to be solved. A major challenge is the scan size. The dimensions of digitized tissue samples may exceed 100,000 by 100,000 pixels causing memory and efficiency obstacles for real-time processing. The main contribution of this work is representing a WSI by selecting a small number of patches for algorithmic processing (e.g., indexing and search). As a result, we r…
A deep learning pipeline for PAM50 subtype classification using histopathology images and multi-objective patch selection
Austrian Center for Medical Innovation and Technology（奥地利医学创新与技术中心）；Danube Private University（多瑙河私立大学）；Medical University of Vienna（维也纳医科大学）；Patho im Zentrum（中心病理机构）；Institute for Systems and Robotics (ISR-Lisboa/LARSyS)（系统与机器人研究所（里斯本系统与机器人研究所/卢西塔尼亚机器人与系统协会））；Instituto Superior Técnico（高等技术学院）；University of Lisbon（里斯本大学）；The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）
There is a strong need for automated systems to improve diagnostic quality and reduce the analysis time in histopathology image processing. Automated detection and classification of pathological tissue characteristics with computer-aided diagnostic systems are a critical step in the early diagnosis and treatment of diseases. Once a pathology image is scanned by a microscope and loaded onto a computer, it can be used for automated detection and classification of diseases. In this study, the DenseNet-161 and ResNet-50 pre-trained CNN models have been used to classify digital histopathology patch…
Advancement in digital pathology and artificial intelligence has enabled deep learning-based computer vision techniques for automated disease diagnosis and prognosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized, making them infeasible to be used directly for training deep neural networks. Hence, for modeling, a two-stage approach is adopted: Patch representations are extracted first, followed by the aggregation for WSI prediction. These approaches require detailed pixel-level annotations for training the patch encoder. However, obtaining thes…
University of Toronto（多伦多大学）；National Cancer Research Centre（国家癌症研究中心）；Institute for Oncology and Radiology（肿瘤与放射学研究所）
Accurate diagnosis of breast cancer in histopathology images is challenging due to the heterogeneity of cancer cell growth as well as of a variety of benign breast tissue proliferative lesions. In this paper, we propose a practical and self-interpretable invasive cancer diagnosis solution. With minimum annotation information, the proposed method mines contrast patterns between normal and malignant images in unsupervised manner and generates a probability map of abnormalities to verify its reasoning. Particularly, a fully convolutional autoencoder is used to learn the dominant structural patter…
The Pennsylvania State University（宾夕法尼亚州立大学）；Lister Hill National Center for Biomedical Communications（利斯特希尔国家生物医学通信中心）；National Library of Medicine（国家医学图书馆）；Penn State Health Milton S. Hershey Medical Center（宾夕法尼亚州立大学健康米尔顿·S·赫尔希医学中心）；Penn State College Of Medicine（宾夕法尼亚州立大学医学院）
Histopathological analysis is the present gold standard for precancerous lesion diagnosis. The goal of automated histopathological classification from digital images requires supervised training, which requires a large number of expert annotations that can be expensive and time-consuming to collect. Meanwhile, accurate classification of image patches cropped from whole-slide images is essential for standard sliding window based histopathology slide classification methods. To mitigate these issues, we propose a carefully designed conditional GAN model, namely HistoGAN, for synthesizing realisti…
Histopathology image analysis can be considered as a Multiple instance learning (MIL) problem, where the whole slide histopathology image (WSI) is regarded as a bag of instances (i.e, patches) and the task is to predict a single class label to the WSI. However, in many real-life applications such as computational pathology, discovering the key instances that trigger the bag label is of great interest because it provides reasons for the decision made by the system. In this paper, we propose a deep convolutional neural network (CNN) model that addresses the primary task of a bag classification o…
Histopathology digital scans are large-size images that contain valuable information at the pixel level. Content-based comparison of these images is a challenging task. This study proposes a content-based similarity measure for high-resolution gigapixel histopathology images. The proposed similarity measure is an expansion of cosine vector similarity to a matrix. Each image is divided into same-size patches with a meaningful amount of information (i.e., contained enough tissue). The similarity is measured by the extraction of patch-level deep embeddings of the last pooling layer of a pre-train…
Breast cancer has the highest mortality among cancers in women. Computer-aided pathology to analyze microscopic histopathology images for diagnosis with an increasing number of breast cancer patients can bring the cost and delays of diagnosis down. Deep learning in histopathology has attracted attention over the last decade of achieving state-of-the-art performance in classification and localization tasks. The convolutional neural network, a deep learning framework, provides remarkable results in tissue images analysis, but lacks in providing interpretation and reasoning behind the decisions. …
Stony Brook Medicine（石溪大学医学中心）；Stony Brook University（石溪大学）；Cold Spring Harbor Laboratory（冷泉港实验室）
Histopathology remains the gold standard for diagnosis of various cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for various tasks, including immune cell detection and microsatellite instability classification. The state-of-the-art for each task often employs base architectures that have been pretrained for image classification on ImageNet. The standard approach to develop classifiers in histopathology tends to focus narrowly on optimizing models for a single task, not considering the aspects of modeling innovatio…
The distribution and appearance of nuclei are essential markers for the diagnosis and study of cancer. Despite the importance of nuclear morphology, there is a lack of large scale, accurate, publicly accessible nucleus segmentation data. To address this, we developed an analysis pipeline that segments nuclei in whole slide tissue images from multiple cancer types with a quality control process. We have generated nucleus segmentation results in 5,060 Whole Slide Tissue images from 10 cancer types in The Cancer Genome Atlas. One key component of our work is that we carried out a multi-level qual…
University of Waterloo（滑铁卢大学）；Amirkabir University of Technology（阿米尔卡比尔理工大学）
We explore the problem of classification within a medical image data-set based on a feature vector extracted from the deepest layer of pre-trained Convolution Neural Networks. We have used feature vectors from several pre-trained structures, including networks with/without transfer learning to evaluate the performance of pre-trained deep features versus CNNs which have been trained by that specific dataset as well as the impact of transfer learning with a small number of samples. All experiments are done on Kimia Path24 dataset which consists of 27,055 histopathology training patches in 24 tis…
Multi-Resolution Histopathology Patch Graphs for Ovarian Cancer Subtyping
Centre for Computational Imaging and Simulation Technologies in Biomedicine (CISTIB)（生物医学计算成像与模拟技术中心）；School of Computing, University of Leeds（利兹大学计算机学院）；Leeds Institute of Medical Research at St James’s, School of Medicine, University of Leeds（利兹大学医学院圣詹姆斯利兹医学研究所）；Leeds Cancer Centre, St James’s University Hospital（圣詹姆斯大学医院利兹癌症中心）
Due to the recent advancements in machine vision, digital pathology has gained significant attention. Histopathology images are distinctly rich in visual information. The tissue glass slide images are utilized for disease diagnosis. Researchers study many methods to process histopathology images and facilitate fast and reliable diagnosis; therefore, the availability of high-quality slides becomes paramount. The quality of the images can be negatively affected when the glass slides are ink-marked by pathologists to delineate regions of interest. As an example, in one of the largest public histo…
Weakly Supervised Spatial Grounding for Discriminative Attention-Based Ultrasound-Histopathology Alignment in Prostate Cancer Grading
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Ordensklinikum Linz（林茨教团医院）；University of Alberta（阿尔伯塔大学）；Exact Imaging（Exact Imaging公司）
Pennsylvania State University（宾夕法尼亚州立大学）；National Library of Medicine, National Institutes of Health（美国国立卫生研究院国家医学图书馆）
Cervical intraepithelial neoplasia (CIN) grade of histopathology images is a crucial indicator in cervical biopsy results. Accurate CIN grading of epithelium regions helps pathologists with precancerous lesion diagnosis and treatment planning. Although an automated CIN grading system has been desired, supervised training of such a system would require a large amount of expert annotations, which are expensive and time-consuming to collect. In this paper, we investigate the CIN grade classification problem on segmented epithelium patches. We propose to use conditional Generative Adversarial Netw…
University of Waterloo（滑铁卢大学）；Huron Digital Pathology（休伦数字病理学公司）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）；Vector Institute（矢量研究所）
With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform milli…
Vector Institute（向量研究所）；University of Waterloo（滑铁卢大学）
Whole slide imaging (WSI) refers to the digitization of a tissue specimen which enables pathologists to explore high-resolution images on a monitor rather than through a microscope. The formation of tissue folds occur during tissue processing. Their presence may not only cause out-of-focus digitization but can also negatively affect the diagnosis in some cases. In this paper, we have compared five pre-trained convolutional neural networks (CNNs) of different depths as feature extractors to characterize tissue folds. We have also explored common classifiers to discriminate folded tissue against…
Helmholtz Munich（亥姆霍兹慕尼黑中心）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In many histopathology tasks, sample classification depends on morphological details in tissue or single cells that are only visible at the highest magnification. For a pathologist, this implies tedious zooming in and out, while for a computational decision support algorithm, it leads to the analysis of a huge number of small image patches per whole slide image (WSI). Attention-based multiple instance learning (MIL), where attention estimation is learned in a weakly supervised manner, has been successfully applied in computational histopathology, but it is challenged by large numbers of irrele…
Automated classification of histopathological whole-slide images (WSI) of breast tissue requires analysis at very high resolutions with a large contextual area. In this paper, we present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, ductal carcinoma in situ (DCIS), and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution patches to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stac…
Low-resource finetuning of foundation models beats state-of-the-art in histopathology
School of Computation and Information Technology, Technical University of Munich（慕尼黑工业大学计算与信息技术学院）；Helmholtz Zentrum München - German Research Center for Environmental Health（慕尼黑亥姆霍兹中心——德国环境健康研究中心）；School of Biomedical Engineering and Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）
University of Liverpool（利物浦大学）；Cixi Institute of Biomedical Engineering, Chinese Academy of Sciences（中国科学院慈溪生物医学工程研究所）；China Science IntelliCloud Technology Co., Ltd（中科智云科技有限公司）；Remark AI UK Limited（Remark AI英国有限公司）
Multiple instance learning (MIL) has been increasingly used in the classification of histopathology whole slide images (WSIs). However, MIL approaches for this specific classification problem still face unique challenges, particularly those related to small sample cohorts. In these, there are limited number of WSI slides (bags), while the resolution of a single WSI is huge, which leads to a large number of patches (instances) cropped from this slide. To address this issue, we propose to virtually enlarge the number of bags by introducing the concept of pseudo-bags, on which a double-tier MIL f…
Faculty of Informatics and Data Science, University of Regensburg, Germany（信息学与数据科学学院，雷根萨大学）；Indian Institute of Information Technology Bhopal, India（比哈尔信息学院）
Stony Brook University（石溪大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Stony Brook University Hospital（石溪大学医院）
Histopathology images are crucial to the study of complex diseases such as cancer. The histologic characteristics of nuclei play a key role in disease diagnosis, prognosis and analysis. In this work, we propose a sparse Convolutional Autoencoder (CAE) for fully unsupervised, simultaneous nucleus detection and feature extraction in histopathology tissue images. Our CAE detects and encodes nuclei in image patches in tissue images into sparse feature maps that encode both the location and appearance of nuclei. Our CAE is the first unsupervised detection network for computer vision applications. T…
PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）；David Geffen School of Medicine, UCLA（加州大学洛杉矶分校大卫·格芬医学院）；Allen Institute for AI (AI2)（艾伦人工智能研究所）
Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Stony Brook University（石溪大学）；Shaukat Khanum Cancer Hospital（肖卡特·卡努姆癌症医院）；Australian National University（澳大利亚国立大学）
Promptable Representation Distribution Learning and Data Augmentation for Gigapixel Histopathology WSI Analysis
Beihang University（北京航空航天大学）；Tianmushan Laboratory（天目山实验室）；Hefei University of Technology（合肥工业大学）；the First Affiliated Hospital of USTC（中国科学技术大学附属第一医院）；University of Science and Technology of China（中国科学技术大学）
SegTME-UNI2: A Foundation Model-Based Framework for Generalisable Multiclass Cell Segmentation and LLM-Driven Tumour Microenvironment Characterisation in Histopathology
Department of Data Science and Artificial Intelligence, School of Computing and Artificial Intelligence, Faculty of Engineering and Technology, Sunway University（双威大学工程与技术学院计算与人工智能学院数据科学与人工智能系）；Faculty of Dentistry, Universiti Malaya（马来亚大学牙科学院）
