Prediction tasks in digital pathology are challenging due to the massive size of whole-slide images (WSIs) and the weak nature of training signals. Advances in computing, data availability, and self-supervised learning (SSL) have paved the way for slide-level foundation models (SLFMs) that can improve prediction tasks in low-data regimes. However, current methods under-utilize shared information between tasks and modalities. To overcome this challenge, we propose ModalTune, a novel fine-tuning framework which introduces the Modal Adapter to integrate new modalities without modifying SLFM weights. Additionally, we use large-language models (LLMs) to encode labels as text, capturing semantic relationships across multiple tasks and cancer types in a single training recipe. ModalTune achieves state-of-the-art (SOTA) results against both uni-modal and multi-modal models across four cancer types, jointly improving survival and cancer subtype prediction while remaining competitive in pan-cancer settings. Additionally, we show ModalTune is generalizable to two out-of-distribution (OOD) datasets. To our knowledge, this is the first unified fine-tuning framework for multi-modal, multi-task, and pan-cancer modeling in digital pathology.
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）
The rapidly emerging field of computational pathology has the potential to enable objective diagnosis, therapeutic response prediction and identification of new morphological features of clinical relevance. However, deep learning-based computational pathology approaches either require manual annotation of gigapixel whole slide images (WSIs) in fully-supervised settings or thousands of WSIs with slide-level labels in a weakly-supervised setting. Moreover, whole slide level computational pathology methods also suffer from domain adaptation and interpretability issues. These challenges have preve…
Pathology slides of lung malignancies are classified using resource-frugal convolution neural networks (CNNs) that may be deployed on mobile devices. In particular, the challenging task of distinguishing adenocarcinoma (LUAD) and squamous-cell carcinoma (LUSC) lung cancer subtypes is approached in two stages. First, whole-slide histopathology images are downsampled to a size too large for CNN analysis but large enough to retain key anatomic detail. The downsampled images are decomposed into smaller square tiles, which are sifted based on their image entropies. A lightweight CNN produces tile-l…
University of Toronto（多伦多大学）；Huron Digital Pathology Inc.（休伦数字病理公司）
One of the challenges facing the adoption of digital pathology workflows for clinical use is the need for automated quality control. As the scanners sometimes determine focus inaccurately, the resultant image blur deteriorates the scanned slide to the point of being unusable. Also, the scanned slide images tend to be extremely large when scanned at greater or equal 20X image resolution. Hence, for digital pathology to be clinically useful, it is necessary to use computational tools to quickly and accurately quantify the image focus quality and determine whether an image needs to be re-scanned.…
Paired Uterine Whole-Slide Images and Pathology Reports for Multimodal Computational Pathology
Institute of Pathology, Technical University of Munich（慕尼黑工业大学病理研究所）；Computer Aided Medical Procedures (CAMP), Technical University of Munich（慕尼黑工业大学计算机辅助医疗程序（CAMP））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Department of Human Pathology, Juntendo University Graduate School of Medicine（顺天堂大学医学研究生院人体病理学部）；The Hong Kong University of Science and Technology（香港科技大学）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
This project aims to break down large pathology images into small tiles and then cluster those tiles into distinct groups without the knowledge of true labels, our analysis shows how difficult certain aspects of clustering tumorous and non-tumorous cells can be and also shows that comparing the results of different unsupervised approaches is not a trivial task. The project also provides a software package to be used by the digital pathology community, that uses some of the approaches developed to perform unsupervised unsupervised tile classification, which could then be easily manually labelle…
Bayer AG（拜耳股份公司）；Median Technologies（Median科技公司）；Bayer Crop Science（拜耳作物科学公司）；Pfizer Worldwide Research Development and Medical（辉瑞全球研发与医疗部门）
In pre-clinical pathology, there is a paradox between the abundance of raw data (whole slide images from many organs of many individual animals) and the lack of pixel-level slide annotations done by pathologists. Due to time constraints and requirements from regulatory authorities, diagnoses are instead stored as slide labels. Weakly supervised training is designed to take advantage of those data, and the trained models can be used by pathologists to rank slides by their probability of containing a given lesion of interest. In this work, we propose a novel contextualized eXplainable AI (XAI) f…
Virchow: A Million-Slide Digital Pathology Foundation Model
Paige；Microsoft Research（微软研究院）；NSW Health Pathology（新南威尔士州卫生病理学）；St George Hospital（圣乔治医院）；Memorial Sloan Kettering Cancer Center（纪念斯隆凯特琳癌症中心）；University of Rochester（罗切斯特大学）
Comparing ImageNet Pre-training with Digital Pathology Foundation Models for Whole Slide Image-Based Survival Analysis
Imperial College London, Department of Electrical and Electronic Engineering, London, UK（帝国理工学院伦敦分校，电子与电气工程系）；Department of Pathology, Institut de Pathologie Multisite, Groupement Hospitalier Sud, Lyon University Hospital, Pierre-Bénite, France（病理学系，多站点病理研究所，南部医院集团，里昂大学医院，皮埃尔-贝内蒂，法国）；University of Lyon, Université Claude Bernard Lyon 1, Lyon, France（里昂大学， Claude Bernard 里昂第一大学，里昂，法国）
WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；School of Engineering, Westlake University（西湖大学工程学院）
F. Hoffmann-La Roche AG（罗氏制药）；Genentech, Inc.（基因泰克公司）
Recently, vision transformers were shown to be capable of outperforming convolutional neural networks when pretrained on sufficient amounts of data. In comparison to convolutional neural networks, vision transformers have a weaker inductive bias and therefore allow a more flexible feature detection. Due to their promising feature detection, this work explores vision transformers for tumor detection in digital pathology whole slide images in four tissue types, and for tissue type identification. We compared the patch-wise classification performance of the vision transformer DeiT-Tiny to the sta…
Multimodal Whole Slide Foundation Model for Pathology
Mass General Brigham, Harvard Medical School（哈佛医学院，麻省总医院布里格姆医疗系统）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Broad Institute of Harvard and MIT（哈佛与麻省理工学院博德研究所）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心——德国环境与健康研究中心）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology（麻省理工学院哈佛-麻省卫生科学与技术分部）；Department of Pathology, Pusan National University（釜山大学病理学系）；Department of Pathology and Laboratory Medicine, Emory University School of Medicine（埃默里大学医学院病理学与检验医学系）；Harvard Data Science Initiative, Harvard University（哈佛大学哈佛数据科学计划）；Department of Systems Biology, Harvard Medical School（哈佛医学院系统生物学系）；Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo（东京大学医学部预防医学系）
Can We Simplify Slide-level Fine-tuning of Pathology Foundation Models?
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；University of Oxford（牛津大学）；The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）；University of Washington（华盛顿大学）
Huron Digital Pathology（休伦数字病理学公司）；University of Waterloo（滑铁卢大学）；Vector Institute（矢量研究所）；McMaster University（麦克马斯特大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
The emergence of digital pathology has opened new horizons for histopathology and cytology. Artificial-intelligence algorithms are able to operate on digitized slides to assist pathologists with diagnostic tasks. Whereas machine learning involving classification and segmentation methods have obvious benefits for image analysis in pathology, image search represents a fundamental shift in computational pathology. Matching the pathology of new patients with already diagnosed and curated cases offers pathologist a novel approach to improve diagnostic accuracy through visual inspection of similar c…
Digital pathology has attracted significant attention in recent years. Analysis of Whole Slide Images (WSIs) is challenging because they are very large, i.e., of Giga-pixel resolution. Identifying Regions of Interest (ROIs) is the first step for pathologists to analyse further the regions of diagnostic interest for cancer detection and other anomalies. In this paper, we investigate the use of RCNN, which is a deep machine learning technique, for detecting such ROIs only using a small number of labelled WSIs for training. For experimentation, we used real WSIs from a public hospital pathology s…
Barco Healthcare（巴可医疗）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
It is our conjecture that the variability of colors in a pathology image effects the interpretation of pathology cases, whether it is diagnostic accuracy, diagnostic confidence, or workflow efficiency. In this paper, digital pathology images are analyzed to quantify the perceived difference in color that occurs due to display aging, in particular a change in the maximum luminance, white point, and color gamut. The digital pathology images studied include diagnostically important features, such as the conspicuity of nuclei. Three different display aging models are applied to images: aging of lu…
Mars Digital Technologies（玛氏数字科技）；Antech Diagnostics（Antech诊断公司）；Stanford University（斯坦福大学）
Background: Histopathology is an important modality for the diagnosis and management of many diseases in modern healthcare, and plays a critical role in cancer care. Pathology samples can be large and require multi-site sampling, leading to upwards of 20 slides for a single tumor, and the human-expert tasks of site selection and and quantitative assessment of mitotic figures are time consuming and subjective. Automating these tasks in the setting of a digital pathology service presents significant opportunities to improve workflow efficiency and augment human experts in practice. Approach: Mul…
High-performance Data Management for Whole Slide Image Analysis in Digital Pathology
Vanderbilt University（范德堡大学）；Vanderbilt University Medical Center（范德堡大学医学中心）；NVIDIA Corporation（英伟达公司）；Georgia State University（佐治亚州立大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Weill Cornell Medicine（威尔康奈尔医学院）
本文提出基于 ADIOS2 的数字病理 WSI 数据管理流水线，通过并行化补丁级 I/O 缩短检索时间，在 CPU 场景获两倍加速，并在 GPU 场景达到与 NVIDIA Magnum IO GDS 相当的性能。
Sorbonne Université（索邦大学）；Institut du Cerveau - Paris Brain Institute - ICM（巴黎大脑研究所（ICM））；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Inserm（法国国家健康与医学研究院）；AP-HP（巴黎公共援助医院）；Hôpital Pitié Salpêtrière（皮蒂耶-萨尔佩特里埃医院）
Non-invasive brain imaging techniques allow understanding the behavior and macro changes in the brain to determine the progress of a disease. However, computational pathology provides a deeper understanding of brain disorders at cellular level, able to consolidate a diagnosis and make the bridge between the medical image and the omics analysis. In traditional histopathology, histology slides are visually inspected, under the microscope, by trained pathologists. This process is time-consuming and labor-intensive; therefore, the emergence of Computational Pathology has triggered great hope to ea…
Deep learning methods are widely applied in digital pathology to address clinical challenges such as prognosis and diagnosis. As one of the most recent applications, deep models have also been used to extract molecular features from whole slide images. Although molecular tests carry rich information, they are often expensive, time-consuming, and require additional tissue to sample. In this paper, we propose tRNAsfomer, an attention-based topology that can learn both to predict the bulk RNA-seq from an image and represent the whole slide image of a glass slide simultaneously. The tRNAsfomer use…
School of Computer Science（计算机科学学院）；School of Medicine（医学院）；University of St Andrews（圣安德鲁斯大学）
Digital pathology (DP) is a new research area which falls under the broad umbrella of health informatics. Owing to its potential for major public health impact, in recent years DP has been attracting much research attention. Nevertheless, a wide breadth of significant conceptual and technical challenges remain, few of them greater than those encountered in the field of oncology. The automatic analysis of digital pathology slides of cancerous tissues is particularly problematic due to the inherent heterogeneity of the disease, extremely large images, amongst numerous others. In this paper we in…
Assessing and Enhancing Robustness of Deep Learning Models with Corruption Emulation in Digital Pathology
College of Engineering, Peking University（北京大学工学院）；Third Hospital, Peking University Health Science Center（北京大学医学部第三医院）；Institute of Biomedical Engineering, Beijing Institute of Collaborative Innovation（北京协同创新研究院生物医学工程研究所）
MOOZY: A Patient-First Foundation Model for Computational Pathology
CSSE, Concordia University, Montr\' e al, Canada Mila -- Qu\' e bec AI Institute, Montr\' e al, Canada CHUM, Universit\' e de Montr\' e al, Montr\' e al, Canada IRIC, Universit\' e de Montr\' e al, Montr\' e al, Canada Universit\' e de Montr\' e al, Montr\' e al, Canada Canada CIFAR Chair, Polytechnique Montr\' e al, Montr\' e al, Canada Dept.\ of Pathology, McGill University, Montr\' e al, Canada
GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis
Microsoft Research（微软研究院）；Paul G. Allen School of Computer Science and Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Providence Genomics（普罗维登斯基因组学公司）；Earle A. Chiles Research Institute, Providence Cancer Institute（普罗维登斯癌症研究所厄尔·A·奇尔斯研究所）；Providence Research Network（普罗维登斯研究网络）
Foundation Models for Slide-level Cancer Subtyping in Digital Pathology
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano (HUMAN-Tech)（以人为本技术大学研究院（HUMAN-Tech））；Universitat Politécnica de Valencia（瓦伦西亚理工大学）；Valencian Graduate School and Research Network for Artificial Intelligence (valgraAI)（瓦伦西亚人工智能研究生院与研究网络（valgraAI））
Northwestern Polytechnical University（西北工业大学）；Tianjin Medical University（天津医科大学）
Goal: Squamous cell carcinoma of cervix is one of the most prevalent cancer worldwide in females. Traditionally, the most indispensable diagnosis of cervix squamous carcinoma is histopathological assessment which is achieved under microscope by pathologist. However, human evaluation of pathology slide is highly depending on the experience of pathologist, thus big inter- and intra-observer variability exists. Digital pathology, in combination with deep learning provides an opportunity to improve the objectivity and efficiency of histopathologic slide analysis. Methods: In this study, we obtaine…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Google Health（谷歌健康）
Metadata are general characteristics of the data in a well-curated and condensed format, and have been proven to be useful for decision making, knowledge discovery, and also heterogeneous data organization of biobank. Among all data types in the biobank, pathology is the key component of the biobank and also serves as the gold standard of diagnosis. To maximize the utility of biobank and allow the rapid progress of biomedical science, it is essential to organize the data with well-populated pathology metadata. However, manual annotation of such information is tedious and time-consuming. In the…
Stony Brook University（石溪大学）；University of Brasília（巴西利亚大学）；Emory University（埃默里大学）
An accurate steatosis quantification with pathology tissue samples is of high clinical importance. However, such pathology measurement is manually made in most clinical practices, subject to severe reader variability due to large sampling bias and poor reproducibility. Although some computerized automated methods are developed to quantify the steatosis regions, they present limited analysis capacity for high resolution whole-slide microscopy images and accurate overlapped steatosis division. In this paper, we propose a method that extracts an individual whole tissue piece at high resolution wi…
Annotating cancerous regions in whole-slide images (WSIs) of pathology samples plays a critical role in clinical diagnosis, biomedical research, and machine learning algorithms development. However, generating exhaustive and accurate annotations is labor-intensive, challenging, and costly. Drawing only coarse and approximate annotations is a much easier task, less costly, and it alleviates pathologists' workload. In this paper, we study the problem of refining these approximate annotations in digital pathology to obtain more accurate ones. Some previous works have explored obtaining machine le…
From slides to AI-ready maps: Standardized multi-layer tissue maps as metadata for artificial intelligence in digital pathology
Medical University of Graz（格拉茨医科大学）；Graz University of Technology（格拉茨工业大学）；University of Maribor（马里博尔大学）；Masaryk University（马萨里克大学）；Masaryk Memorial Cancer Institute（马萨里克纪念癌症研究所）；University of Natural Resources and Life Sciences Vienna（维也纳自然资源与生命科学大学）；BBMRI-ERIC（欧洲生物样本库与生物分子资源研究基础设施联盟）；University Hospital RWTH Aachen（亚琛工业大学医院）；Charité - Universitätsmedizin Berlin（柏林夏里特医学院）；Freie Universität Berlin（柏林自由大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）；EMPAIA International e.V.（EMPAIA国际协会）；European Society of Digital and Integrative Pathology（欧洲数字与整合病理学会）
Semantic Mosaicing of Histo-Pathology Image Fragments using Visual Foundation Models
Computational Imaging Research Lab（计算成像研究实验室）；Division of General and Pediatric Radiology（普通放射科与小儿放射科 division）；Christian Doppler Laboratory for Machine Learning Driven Precision Imaging（Christian Doppler 机器学习驱动的精准成像实验室）；Comprehensive Center for Artificial Intelligence in Medicine（医学人工智能综合中心）；Department of Pathology（病理学系）
From Whole-slide Image to Biomarker Prediction: A Protocol for End-to-End Deep Learning in Computational Pathology
Technical University Dresden（德累斯顿工业大学）；University of St Andrews（圣安德鲁斯大学）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心-德国环境与健康研究中心）；Technical University of Munich（慕尼黑工业大学）；University Hospital Aachen（亚琛大学医院）；University Medical Center Mainz（美因茨大学医学中心）；University of Leeds（利兹大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心 (NCT)）；University Hospital Heidelberg（海德堡大学医院）；Technische Universität Dresden（德累斯顿工业大学）
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Tsinghua University（清华大学）；McMaster University（麦克马斯特大学）；Shanghai Jiao Tong University（上海交通大学）
Whole slide imaging (WSI) is an emerging technology for digital pathology. The process of autofocusing is the main influence of the performance of WSI. Traditional autofocusing methods either are time-consuming due to repetitive mechanical motions, or require additional hardware and thus are not compatible to current WSI systems. In this paper, we propose the concept of \textit{virtual autofocusing}, which does not rely on mechanical adjustment to conduct refocusing but instead recovers in-focus images in an offline learning-based manner. With the initial focal position, we only perform two-sh…
Stony Brook University（石溪大学）；Brigham and Women's Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Fraunhofer Institute for Medical Image Computing MEVIS（弗劳恩霍夫医学图像计算研究所）；University of Bremen（不来梅大学）
Histopathological assessments, including surgical resection and core needle biopsy, are the standard procedures in the diagnosis of the prostate cancer. Current interpretation of the histopathology images includes the determination of the tumor area, Gleason grading, and identification of certain prognosis-critical features. Such a process is not only tedious, but also prune to intra/inter-observe variabilities. Recently, FDA cleared the marketing of the first whole slide imaging system for digital pathology. This opens a new era for the computer aided prostate image analysis and feature extra…
University of Texas Health Science Center at Houston（德克萨斯大学休斯顿健康科学中心）
Recent studies have shown promising results in using Deep Learning to detect malignancy in whole slide imaging. However, they were limited to just predicting positive or negative finding for a specific neoplasm. We attempted to use Deep Learning with a convolutional neural network algorithm to build a lymphoma diagnostic model for four diagnostic categories: benign lymph node, diffuse large B cell lymphoma, Burkitt lymphoma, and small lymphocytic lymphoma. Our software was written in Python language. We obtained digital whole slide images of Hematoxylin and Eosin stained slides of 128 cases in…
Histopathology images; microscopy images of stained tissue biopsies contain fundamental prognostic information that forms the foundation of pathological analysis and diagnostic medicine. However, diagnostics from histopathology images generally rely on a visual cognitive assessment of tissue slides which implies an inherent element of interpretation and hence subjectivity. Access to digitized histopathology images enabled the development of computational systems aiming at reducing manual intervention and automating parts of pathologists' workflow. Specifically, applications of deep learning to…
Ensemble learning of pathology foundation models for precision oncology
Department of Radiation Oncology, Stanford University School of Medicine（放射肿瘤科，斯坦福大学医学院）；Department of Pathology, Stanford University School of Medicine（病理学系，斯坦福大学医学院）；Department of Medicine (Oncology), Stanford University School of Medicine（医学系（肿瘤学），斯坦福大学医学院）；Department of Neurosurgery, Stanford University School of Medicine（神经外科，斯坦福大学医学院）；Stanford Institute for Human-Centered Artificial Intelligence（斯坦福大学人本人工智能研究所）
The PAR dataset: Prostate biopsy whole slide images from an underrepresented Middle Eastern population
Koya University（科亚大学）；Karolinska Institutet（卡罗林斯卡学院）；Hawler Medical University（哈勒医科大学）；PAR Private Hospital（PAR私立医院）；University of Sulaimani（苏莱曼尼亚大学）；Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Turku（图尔库大学）
Due to the recent advancements in machine vision, digital pathology has gained significant attention. Histopathology images are distinctly rich in visual information. The tissue glass slide images are utilized for disease diagnosis. Researchers study many methods to process histopathology images and facilitate fast and reliable diagnosis; therefore, the availability of high-quality slides becomes paramount. The quality of the images can be negatively affected when the glass slides are ink-marked by pathologists to delineate regions of interest. As an example, in one of the largest public histo…
