A Clinical Benchmark of Public Self-Supervised Pathology Foundation Models
Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；Hasso Plattner Institute at Mount Sinai（西奈山哈索·普拉特纳研究院）
Gabriele Campanella, Shengjia Chen, Ruchika Verma, Jennifer Zeng, Aryeh Stock, Matt Croken, Brandon Veremis, Abdulkadir Elmas, Kuan-lin Huang, Ricky Kwan, Jane … 展开作者
Gabriele Campanella, Shengjia Chen, Ruchika Verma, Jennifer Zeng, Aryeh Stock, Matt Croken, Brandon Veremis, Abdulkadir Elmas, Kuan-lin Huang, Ricky Kwan, Jane Houldsworth, Adam J. Schoenfeld, Chad Vanderbilt
备注：
arXiv admin note: text overlap with arXiv:2310.07033
The use of self-supervised learning (SSL) to train pathology foundation models has increased substantially in the past few years. Notably, several models trained on large quantities of clinical data have been made publicly available in recent months. This will significantly enhance scientific research in computational pathology and help bridge the gap between research and clinical deployment. With the increase in availability of public foundation models of different sizes, trained using different algorithms on different datasets, it becomes important to establish a benchmark to compare the performance of such models on a variety of clinically relevant tasks spanning multiple organs and diseases. In this work, we present a collection of pathology datasets comprising clinical slides associated with clinically relevant endpoints including cancer diagnoses and a variety of biomarkers generated during standard hospital operation from two medical centers. We leverage these datasets to systematically assess the performance of public pathology foundation models and provide insights into best practices for training new foundation models and selecting appropriate pretrained models.
PathOrchestra: A Comprehensive Foundation Model for Computational Pathology with Over 100 Diverse Clinical-Grade Tasks
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fourth Military Medical University（第四军医大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；SenseTime Research（商汤科技研究院）；Hong Kong University of Science and Technology（香港科技大学）；Peking University（北京大学）；Shengqiang Technology Co. Ltd.（胜强科技有限公司）；Institute of Materials Research, Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院材料研究院）
Pathryoshka: Compressing Pathology Foundation Models via Multi-Teacher Knowledge Distillation with Nested Embeddings
Technical University of Munich（慕尼黑技术大学）；Helmholtz Munich（马克斯·普朗克研究所慕尼黑分部）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
The Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理学重点实验室）；Jinfeng Laboratory（金凤实验室）；Shandong First Medical University（山东第一医科大学）；Shandong Provincial Qianfoshan Hospital（山东省千佛山医院）；The Third Affiliated Hospital of Kunming Medical University（昆明医科大学第三附属医院）；Yunnan Cancer Hospital（云南省肿瘤医院）；The Chinese University of Hong Kong（香港中文大学）；Prince of Wales Hospital（威尔士亲王医院）；Sun Yat-sen University（中山大学）
PathBench: A comprehensive comparison benchmark for pathology foundation models towards precision oncology
Hong Kong University of Science and Technology（香港科技大学）；Fourth Military Medical University（第四军医大学）；Chinese University of Hong Kong（香港中文大学）；Harvard University（哈佛大学）
CORA: Generalizable coronary artery disease assessment and risk stratification from coronary CT angiography using pathology-centric representation learning
Department of Radiology, Northwestern University（西北大学放射学系）；Department of Biomedical Engineering, Northwestern University（西北大学生物医学工程系）；Department of Cardiology, Northwestern University（西北大学心脏病学系）；Department of Preventive Medicine, Northwestern University（西北大学预防医学系）
Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation
The Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理学重点实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The First Affiliated Hospital of Shandong First Medical University and Shandong Provincial Qianfoshan Hospital（山东第一医科大学第一附属医院暨山东省千佛山医院）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）
Benchmarking Pathology Foundation Models for Breast Cancer Survival Prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Division of Oncology, Department of Clinical Sciences Lund, Lund University（卢德大学临床科学系肿瘤学系）；Oxford Suzhou Centre for Advanced Research, University of Oxford（牛津大学苏州先进研究中心）
Department of Pathology and Laboratory Medicine, University of Pennsylvania（病理学与实验室医学系，宾夕法尼亚大学）；Department of Biostatistics, Epidemiology & Informatics, University of Pennsylvania（生物统计学、流行病学与信息学系，宾夕法尼亚大学）；Department of Pathology, University of California at San Francisco（病理学系，加州大学旧金山分校）；Department of Pathology, Stanford University（病理学系，斯坦福大学）；Department of Electrical and System Engineering, University of Pennsylvania（电气与系统工程系，宾夕法尼亚大学）
A General-Purpose Self-Supervised Model for Computational Pathology
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Broad Institute of Harvard and MIT（哈佛-麻省理工博德研究所）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Massachusetts Institute of Technology（麻省理工学院）；Harvard John A. Paulson School of Engineering And Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Harvard University（哈佛大学）
University of Warwick（华威大学）；University of Birmingham（伯明翰大学）；University Hospitals Coventry & Warwickshire（考文垂和沃里克郡大学医院）；The Alan Turing Institute（艾伦·图灵研究所）
Computational Pathology (CPath) is an emerging field concerned with the study of tissue pathology via computational algorithms for the processing and analysis of digitized high-resolution images of tissue slides. Recent deep learning based developments in CPath have successfully leveraged sheer volume of raw pixel data in histology images for predicting target parameters in the domains of diagnostics, prognostics, treatment sensitivity and patient stratification -- heralding the promise of a new data-driven AI era for both histopathology and oncology. With data serving as the fuel and AI as th…
Microsoft Research（微软研究院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Minnesota（明尼苏达大学）；Stanford University（斯坦福大学）；Duke University（杜克大学）；New York University（纽约大学）
Improving speed and image quality of Magnetic Resonance Imaging (MRI) via novel reconstruction approaches remains one of the highest impact applications for deep learning in medical imaging. The fastMRI dataset, unique in that it contains large volumes of raw MRI data, has enabled significant advances in accelerating MRI using deep learning-based reconstruction methods. While the impact of the fastMRI dataset on the field of medical imaging is unquestioned, the dataset currently lacks clinical expert pathology annotations, critical to addressing clinically relevant reconstruction frameworks an…
Artificial Intelligence for Digital and Computational Pathology
Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Broad Institute of Harvard and MIT（哈佛大学与麻省理工学院博德研究所）；Dana-Farber Cancer Institute（丹娜-法伯癌症研究所）；Massachusetts Institute of Technology（麻省理工学院）
DiagR1: A Vision-Language Model Trained via Reinforcement Learning for Digestive Pathology Diagnosis
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Department of Pathology, Liuzhou People’s Hospital Affiliated to Guangxi Medical University（广西医科大学柳州市人民医院病理科）；Department of Immunology, College of Basic Medical Sciences, China Medical University（中国医科大学基础医学学院免疫科）；Greater Bay Area Center for Medical Device Evaluation and Inspection.NMPA（粤港澳大湾区医疗器械评价和检验中心.NMPA）；Shenzhen Shengqiang Technology Co., Ltd.（深圳盛强科技有限公司）；Department of Pathology, Chongqing University Affiliated Three Gorges Hospital（重庆大学附属第三人民医院病理科）
Machine Learning has been applied to pathology images in research and clinical practice with promising outcomes. However, standard ML models often lack the rigorous evaluation required for clinical decisions. Machine learning techniques for natural images are ill-equipped to deal with pathology images that are significantly large and noisy, require expensive labeling, are hard to interpret, and are susceptible to spurious correlations. We propose a set of practical guidelines for ML evaluation in pathology that address the above concerns. The paper includes measures for setting up the evaluati…
Computational pathology is a field that has complemented various subspecialties of diagnostic pathology over the last few years. In this article a brief analyzis the different applications in nephrology is developed. To begin, an overview of the different forms of image production is provided. To continue, the most frequent applications of computer vision models, the salient features of the different clinical applications, and the data protection considerations encountered are described. To finish the development, I delve into the interpretability of these applications, expanding in depth on t…
The University of Tokyo（东京大学）；LPIXEL Inc.（LPIXEL公司）；National Institute of Informatics（国立信息学研究所）；University of Cambridge（剑桥大学）；Keio University（庆应义塾大学）
Despite the success of Convolutional Neural Network-based Computer-Aided Diagnosis research, its clinical applications remain challenging. Accordingly, developing medical Artificial Intelligence (AI) fitting into a clinical environment requires identifying/bridging the gap between AI and Healthcare sides. Since the biggest problem in Medical Imaging lies in data paucity, confirming the clinical relevance for diagnosis of research-proven image augmentation techniques is essential. Therefore, we hold a clinically valuable AI-envisioning workshop among Japanese Medical Imaging experts, physicians…
A Clinically Validated Foundation Model for Comprehensive Lung Pathology Interpretation
Department of Pathology, Nanfang Hospital, Southern Medical University, Guangzhou, China（南方医科大学南芳医院病理科，广州，中国）；Department of Pathology, School of Basic Medical Sciences, Southern Medical University, Guangzhou, China（南方医科大学基础医学学院病理科，广州，中国）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, China（香港科技大学计算机科学与工程系，香港，中国）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology, Guangzhou, China（广东省分子肿瘤病理重点实验室，广州，中国）；Department of Pathology, Shandong Provincial Qianfoshan Hospital, Jinan, Shandong, China（山东省青岛坊山医院病理科，济南，山东，中国）
Hunan University（湖南大学）；Central South University（中南大学）；Third Xiangya Hospital（湘雅三医院）
Computational pathology is part of precision oncology medicine. The integration of high-throughput data including genomics, transcriptomics, proteomics, metabolomics, pathomics, and radiomics into clinical practice improves cancer treatment plans, treatment cycles, and cure rates, and helps doctors open up innovative approaches to patient prognosis. In the past decade, rapid advances in artificial intelligence, chip design and manufacturing, and mobile computing have facilitated research in computational pathology and have the potential to provide better-integrated solutions for whole-slide im…
Multimodal Whole Slide Foundation Model for Pathology
Mass General Brigham, Harvard Medical School（哈佛医学院，麻省总医院布里格姆医疗系统）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Broad Institute of Harvard and MIT（哈佛与麻省理工学院博德研究所）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心——德国环境与健康研究中心）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology（麻省理工学院哈佛-麻省卫生科学与技术分部）；Department of Pathology, Pusan National University（釜山大学病理学系）；Department of Pathology and Laboratory Medicine, Emory University School of Medicine（埃默里大学医学院病理学与检验医学系）；Harvard Data Science Initiative, Harvard University（哈佛大学哈佛数据科学计划）；Department of Systems Biology, Harvard Medical School（哈佛医学院系统生物学系）；Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo（东京大学医学部预防医学系）
MICS Laboratory, CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学中央理工高等电力学院MICS实验室）；Department of Radiotherapy, Gustave Roussy Cancer Campus（古斯塔夫·鲁西癌症园区放射治疗科）；Therapanacea（Therapanacea公司）
The examination of histopathology images is considered to be the gold standard for the diagnosis and stratification of cancer patients. A key challenge in the analysis of such images is their size, which can run into the gigapixels and can require tedious screening by clinicians. With the recent advances in computational medicine, automatic tools have been proposed to assist clinicians in their everyday practice. Such tools typically process these large images by slicing them into tiles that can then be encoded and utilized for different clinical models. In this study, we propose a novel gener…
Deep learning (DL) has shown great potential in digital pathology applications. The robustness of a diagnostic DL-based solution is essential for safe clinical deployment. In this work we evaluate if adding uncertainty estimates for DL predictions in digital pathology could result in increased value for the clinical applications, by boosting the general predictive performance or by detecting mispredictions. We compare the effectiveness of model-integrated methods (MC dropout and Deep ensembles) with a model-agnostic approach (Test time augmentation, TTA). Moreover, four uncertainty metrics are…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）
The rapidly emerging field of computational pathology has the potential to enable objective diagnosis, therapeutic response prediction and identification of new morphological features of clinical relevance. However, deep learning-based computational pathology approaches either require manual annotation of gigapixel whole slide images (WSIs) in fully-supervised settings or thousands of WSIs with slide-level labels in a weakly-supervised setting. Moreover, whole slide level computational pathology methods also suffer from domain adaptation and interpretability issues. These challenges have preve…
Central South University（中南大学）；Shaanxi Provincial People’s Hospital（陕西省人民医院）；Xiangya Hospital, Central South University（中南大学湘雅医院）
[Purpose] The pathology is decisive for disease diagnosis, but relies heavily on the experienced pathologists. Recently, pathological artificial intelligence (PAI) is thought to improve diagnostic accuracy and efficiency. However, the high performance of PAI based on deep learning in the laboratory generally cannot be reproduced in the clinic. [Methods] Because the data preparation is important for PAI, the paper has reviewed PAI-related studies in the PubMed database published from January 2017 to February 2022, and 118 studies were included. The in-depth analysis of methods for preparing dat…
Computational Pathology: A Survey Review and The Way Forward
Qualcomm AI Research（高通人工智能研究院）；Qualcomm Technologies Netherlands B.V.（高通科技荷兰有限公司）；Institute for Research in Immunology and Cancer of the University of Montreal（蒙特利尔大学免疫与癌症研究所）；University of Toronto（多伦多大学）
Towards Robust Foundation Models for Digital Pathology
Berlin Institute for the Foundations of Learning and Data (BIFOLD)（柏林学习与数据基础研究所）；Aignostics GmbH（Aignostics公司）；The Netherlands Cancer Institute Amsterdam (NKI)（荷兰癌症研究所）；Antoni van Leeuwenhoek Hospital (AvL)（安东尼·范·列文虎克医院）；Ludwig Maximilian University（慕尼黑大学）；German Cancer Research Center（德国癌症研究中心）；Korea University（高丽大学）；Max-Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Deep learning models are routinely employed in computational pathology (CPath) for solving problems of diagnostic and prognostic significance. Typically, the generalization performance of CPath models is analyzed using evaluation protocols such as cross-validation and testing on multi-centric cohorts. However, to ensure that such CPath solutions are robust and safe for use in a clinical setting, a critical analysis of their predictive performance and vulnerability to adversarial attacks is required, which is the focus of this paper. Specifically, we show that a highly accurate model for classi…
Assessing and Enhancing Robustness of Deep Learning Models with Corruption Emulation in Digital Pathology
College of Engineering, Peking University（北京大学工学院）；Third Hospital, Peking University Health Science Center（北京大学医学部第三医院）；Institute of Biomedical Engineering, Beijing Institute of Collaborative Innovation（北京协同创新研究院生物医学工程研究所）
University of Edinburgh（爱丁堡大学）；University of California（加利福尼亚大学）；Cedars-Sinai Medical Center（西达赛奈医疗中心）；University of Essex（埃塞克斯大学）；The Alan Turing Institute（艾伦·图灵研究所）
Automated pathology segmentation remains a valuable diagnostic tool in clinical practice. However, collecting training data is challenging. Semi-supervised approaches by combining labelled and unlabelled data can offer a solution to data scarcity. An approach to semi-supervised learning relies on reconstruction objectives (as self-supervision objectives) that learns in a joint fashion suitable representations for the task. Here, we propose Anatomy-Pathology Disentanglement Network (APD-Net), a pathology segmentation model that attempts to learn jointly for the first time: disentanglement of an…
Martinos Center for Biomedical Imaging（马蒂诺斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；MGH & BWH Center for Clinical Data Science（麻省总医院与布莱根妇女医院临床数据科学中心）；Mass General Brigham（麻省医疗布列根）；Isomics, Inc.（Isomics公司）；Center for Integrated Diagnostics（综合诊断中心）；Harvard Medical School（哈佛医学院）；PixelMed Publishing, LLC.（PixelMed出版有限责任公司）；Surgical Planning Laboratory（外科规划实验室）；Brigham and Women’s Hospital（布莱根妇女医院）
Machine learning is revolutionizing image-based diagnostics in pathology and radiology. ML models have shown promising results in research settings, but their lack of interoperability has been a major barrier for clinical integration and evaluation. The DICOM a standard specifies Information Object Definitions and Services for the representation and communication of digital images and related information, including image-derived annotations and analysis results. However, the complexity of the standard represents an obstacle for its adoption in the ML community and creates a need for software l…
HistoEncoder: a digital pathology foundation model for prostate cancer
University of Helsinki（赫尔辛基大学）；Helsinki University Hospital（赫尔辛基大学医院）；Skåne University Hospital（斯科讷大学医院）；Institute for Molecular Medicine Finland (FIMM), HiLIFE, University of Helsinki（赫尔辛基大学HiLIFE项目下属芬兰分子医学研究所（FIMM））；iCAN Digital Precision Cancer Medicine Flagship（iCAN数字精准癌症医学旗舰项目）
Rethinking Clinical Relevance in Chest X-ray Machine Learning: How Evaluation References Define Performance
University of Cambridge（剑桥大学）；Medical University of Vienna（维也纳医科大学）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院 NHS 基金会信托）；King's College London（伦敦国王学院）
Medical Center – University of Freiburg（弗莱堡大学医学中心）；University of Freiburg（弗莱堡大学）；Machine Learning Lab（机器学习实验室）；Neuromedical AI Lab（神经医学人工智能实验室）；Neurorobotics Lab（神经机器人学实验室）
Machine learning (ML) methods have the potential to automate clinical EEG analysis. They can be categorized into feature-based (with handcrafted features), and end-to-end approaches (with learned features). Previous studies on EEG pathology decoding have typically analyzed a limited number of features, decoders, or both. For a I) more elaborate feature-based EEG analysis, and II) in-depth comparisons of both approaches, here we first develop a comprehensive feature-based framework, and then compare this framework to state-of-the-art end-to-end methods. To this aim, we apply the proposed featur…
Physical Color Calibration of Digital Pathology Scanners for Robust Artificial Intelligence Assisted Cancer Diagnosis
Karolinska Institutet（卡罗林斯卡学院）；FFEI Ltd（FFEI有限公司）；University of Turku（图尔库大学）；Stavanger University Hospital（斯塔万格大学医院）；University of Stavanger（斯塔万格大学）；Aarhus University Hospital（奥胡斯大学医院）；Aarhus University（奥胡斯大学）；Tampere University（坦佩雷大学）
