University of Liverpool（利物浦大学）；Cixi Institute of Biomedical Engineering, Chinese Academy of Sciences（中国科学院慈溪生物医学工程研究所）；China Science IntelliCloud Technology Co., Ltd（中科智云科技有限公司）；Remark AI UK Limited（Remark AI英国有限公司）
Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E. Coupland, Yalin Zheng
Multiple instance learning (MIL) has been increasingly used in the classification of histopathology whole slide images (WSIs). However, MIL approaches for this specific classification problem still face unique challenges, particularly those related to small sample cohorts. In these, there are limited number of WSI slides (bags), while the resolution of a single WSI is huge, which leads to a large number of patches (instances) cropped from this slide. To address this issue, we propose to virtually enlarge the number of bags by introducing the concept of pseudo-bags, on which a double-tier MIL framework is built to effectively use the intrinsic features. Besides, we also contribute to deriving the instance probability under the framework of attention-based MIL, and utilize the derivation to help construct and analyze the proposed framework. The proposed method outperforms other latest methods on the CAMELYON-16 by substantially large margins, and is also better in performance on the TCGA lung cancer dataset. The proposed framework is ready to be extended for wider MIL applications. The code is available at: https://github.com/hrzhang1123/DTFD-MIL
xMIL: Insightful Explanations for Multiple Instance Learning in Histopathology
Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Machine Learning Group, Technische Universität Berlin（柏林工业大学机器学习组）；Aignostics GmbH（艾格诺斯蒂克斯有限公司）；Institute of Pathology, Ludwig Maximilian University（路德维希-马克西米利安大学病理学研究所）；German Cancer Research Center（德国癌症研究中心）；Institute of Pathology, Charité Universitätsmedizin（夏里特医学院病理学研究所）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）；Max-Planck Institute for Informatics（马克斯·普朗克信息学研究所）
National Institute of Standards and Technology（国家标准与技术研究院）；Rice University（里士满大学）；Department of Physics, Colorado State University（科罗拉多州立大学物理系）；Electrical Engineering Department, University of Colorado（科罗拉多大学电气工程系）
Helmholtz Munich（亥姆霍兹慕尼黑中心）；Technical University of Munich（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In many histopathology tasks, sample classification depends on morphological details in tissue or single cells that are only visible at the highest magnification. For a pathologist, this implies tedious zooming in and out, while for a computational decision support algorithm, it leads to the analysis of a huge number of small image patches per whole slide image (WSI). Attention-based multiple instance learning (MIL), where attention estimation is learned in a weakly supervised manner, has been successfully applied in computational histopathology, but it is challenged by large numbers of irrele…
Weakly Supervised Contrastive Learning for Histopathology Patch Embeddings
Department of Electrical and Computer Engineering, University of Utah（电气与计算机工程系，犹他大学）；Scientific Computing and Imaging Institute, University of Utah（科学计算与成像研究所，犹他大学）；Department of Pathology, University of Utah（病理学系，犹他大学）
Microsoft Research（微软研究院）；Beihang University（北京航空航天大学）；Tsinghua University（清华大学）
In this paper, we develop a new weakly-supervised learning algorithm to learn to segment cancerous regions in histopathology images. Our work is under a multiple instance learning framework (MIL) with a new formulation, deep weak supervision (DWS); we also propose an effective way to introduce constraints to our neural networks to assist the learning process. The contributions of our algorithm are threefold: (1) We build an end-to-end learning system that segments cancerous regions with fully convolutional networks (FCN) in which image-to-image weakly-supervised learning is performed. (2) We d…
Hispathological image segmentation algorithms play a critical role in computer aided diagnosis technology. The development of weakly supervised segmentation algorithm alleviates the problem of medical image annotation that it is time-consuming and labor-intensive. As a subset of weakly supervised learning, Multiple Instance Learning (MIL) has been proven to be effective in segmentation. However, there is a lack of related information between instances in MIL, which limits the further improvement of segmentation performance. In this paper, we propose a novel weakly supervised method for pixel-l…
Histopathology image analysis can be considered as a Multiple instance learning (MIL) problem, where the whole slide histopathology image (WSI) is regarded as a bag of instances (i.e, patches) and the task is to predict a single class label to the WSI. However, in many real-life applications such as computational pathology, discovering the key instances that trigger the bag label is of great interest because it provides reasons for the decision made by the system. In this paper, we propose a deep convolutional neural network (CNN) model that addresses the primary task of a bag classification o…
Label-free Concept Based Multiple Instance Learning for Gigapixel Histopathology
University of Tübingen（蒂宾根大学）；University of Lucerne（卢塞恩大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Institut du Cancer de l’Ouest（西部癌症研究所）；Oncode Institute（Oncode研究所）
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）；Vector Institute（矢量研究院）
Recent advances in digital pathology have led to the need for Histopathology Image Retrieval (HIR) systems that search through databases of biopsy images to find similar cases to a given query image. These HIR systems allow pathologists to effortlessly and efficiently access thousands of previously diagnosed cases in order to exploit the knowledge in the corresponding pathology reports. Since HIR systems may have to deal with millions of gigapixel images, the extraction of compact and expressive image features must be available to allow for efficient and accurate retrieval. In this paper, we p…
Breast cancer has the highest mortality among cancers in women. Computer-aided pathology to analyze microscopic histopathology images for diagnosis with an increasing number of breast cancer patients can bring the cost and delays of diagnosis down. Deep learning in histopathology has attracted attention over the last decade of achieving state-of-the-art performance in classification and localization tasks. The convolutional neural network, a deep learning framework, provides remarkable results in tissue images analysis, but lacks in providing interpretation and reasoning behind the decisions. …
University of Leeds（利兹大学）；Leeds Teaching Hospitals NHS Trust（利兹教学医院NHS信托基金会）
Weakly-supervised classification of histopathology slides is a computationally intensive task, with a typical whole slide image (WSI) containing billions of pixels to process. We propose Discriminative Region Active Sampling for Multiple Instance Learning (DRAS-MIL), a computationally efficient slide classification method using attention scores to focus sampling on highly discriminative regions. We apply this to the diagnosis of ovarian cancer histological subtypes, which is an essential part of the patient care pathway as different subtypes have different genetic and molecular profiles, treat…
DA-SSL: self-supervised domain adaptor to leverage foundational models in turbt histopathology slides
National Cancer Institute, National Institutes of Health（美国国立卫生研究院国家癌症研究所）；Fred Hutchinson Cancer Center（弗雷德·哈钦森癌症中心）；University of California, San Francisco（加州大学旧金山分校）；Yale University（耶鲁大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Washington（华盛顿大学）
Multiple Instance Learning Framework with Masked Hard Instance Mining for Gigapixel Histopathology Image Analysis
Hong Kong University of Science and Technology（香港科学与技术大学）；Hefei University of Technology（合肥工业大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Promptable Representation Distribution Learning and Data Augmentation for Gigapixel Histopathology WSI Analysis
Beihang University（北京航空航天大学）；Tianmushan Laboratory（天目山实验室）；Hefei University of Technology（合肥工业大学）；the First Affiliated Hospital of USTC（中国科学技术大学附属第一医院）；University of Science and Technology of China（中国科学技术大学）
A Generative Foundation Model for Multimodal Histopathology
Department of Radiation Oncology, Stanford University School of Medicine（斯坦福大学医学院放射肿瘤学系）；Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine（斯坦福大学医学院精神病学与行为科学系）；Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）；Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系）；Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系）；Department of Radiology, Stanford University（斯坦福大学放射学系）；Center for Artificial Intelligence in Medicine and Imaging, Stanford University（斯坦福大学医学与影像人工智能中心）
A Chain-of-thought Reasoning Breast Ultrasound Dataset Covering All Histopathology Categories
Peking University（北京大学）；Peking Union Medical College Hospital（北京大学人民医院）；Peking University Cancer Hospital & Institute（北京大学肿瘤医院）；National Cancer Center / National Clinical Research Center for Cancer / Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院国家癌症中心/国家癌症临床科研中心/癌症医院）；Shenzhen Maternity & Child Health Care Hospital（深圳妇幼保健医院）；Xi’an Jiaotong University（西安交通大学）；Yizhun Medical AI Co., Ltd（亿众医疗人工智能有限公司）
Benchmarking Histopathology Foundation Models for Ovarian Cancer Bevacizumab Treatment Response Prediction from Whole Slide Images
Faculty of Science, University of British Columbia（不列颠哥伦比亚大学理学院）；School of Biomedical Engineering, University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）；Department of Pathology and Laboratory Medicine, University of British Columbia（不列颠哥伦比亚大学病理学与实验医学系）；Canada’s Michael Smith Genome Sciences Centre, BC Cancer Research Institute（加拿大迈克尔·史密斯基因组科学中心，BC癌症研究所）
University of Waterloo（滑铁卢大学）；Huron Digital Pathology（休伦数字病理学公司）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）；Vector Institute（矢量研究所）
With the emergence of digital pathology, searching for similar images in large archives has gained considerable attention. Image retrieval can provide pathologists with unprecedented access to the evidence embodied in already diagnosed and treated cases from the past. This paper proposes a search engine specialized for digital pathology, called Yottixel, a portmanteau for "one yotta pixel," alluding to the big-data nature of histopathology images. The most impressive characteristic of Yottixel is its ability to represent whole slide images (WSIs) in a compact manner. Yottixel can perform milli…
M4: Multi-Proxy Multi-Gate Mixture of Experts Network for Multiple Instance Learning in Histopathology Image Analysis
Hangzhou Institute of Medicine (HIM), Chinese Academy of Sciences（中国科学院杭州医学研究所）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；College of Electrical and Information Engineering, Hunan University（湖南大学电气与信息工程学院）
Despite the promise of Convolutional neural network (CNN) based classification models for histopathological images, it is infeasible to quantify its uncertainties. Moreover, CNNs may suffer from overfitting when the data is biased. We show that Bayesian-CNN can overcome these limitations by regularizing automatically and by quantifying the uncertainty. We have developed a novel technique to utilize the uncertainties provided by the Bayesian-CNN that significantly improves the performance on a large fraction of the test data (about 6% improvement in accuracy on 77% of test data). Further, we pr…
University of Connecticut（康涅狄格大学）；University of Connecticut Health Center（康涅狄格大学健康中心）
Prostate cancer is the most common cancer in men worldwide and the second leading cause of cancer death in the United States. One of the prognostic features in prostate cancer is the Gleason grading of histopathology images. The Gleason grade is assigned based on tumor architecture on Hematoxylin and Eosin (H&E) stained whole slide images (WSI) by the pathologists. This process is time-consuming and has known interobserver variability. In the past few years, deep learning algorithms have been used to analyze histopathology images, delivering promising results for grading prostate cancer. Howev…
Huron Digital Pathology（休伦数字病理学公司）；University of Waterloo（滑铁卢大学）；Vector Institute（矢量研究所）；McMaster University（麦克马斯特大学）；University of Pittsburgh Medical Center（匹兹堡大学医学中心）
The emergence of digital pathology has opened new horizons for histopathology and cytology. Artificial-intelligence algorithms are able to operate on digitized slides to assist pathologists with diagnostic tasks. Whereas machine learning involving classification and segmentation methods have obvious benefits for image analysis in pathology, image search represents a fundamental shift in computational pathology. Matching the pathology of new patients with already diagnosed and curated cases offers pathologist a novel approach to improve diagnostic accuracy through visual inspection of similar c…
Pediatric brain tumor classification using digital histopathology and deep learning: evaluation of SOTA methods on a multi-center Swedish cohort
Department of Biomedical Engineering, Linköping University, Sweden（生物医学工程系，利厄普大学）；Center for Medical Image Science and Visualization, Linköping University, Sweden（医学影像科学与可视化中心，利厄普大学）；Crown Princess Victoria Children’s Hospital and Department of Health, Medicine and Caring Sciences, Linköping University, Sweden（维多利亚公主儿童医院和健康、医学与关怀科学系，利厄普大学）；Department of Radiology and Department of Health, Medicine and Caring Sciences, Linköping University, Sweden（放射科和健康、医学与关怀科学系，利厄普大学）；Department of Oncology-Pathology, Karolinska Institutet, Solna, Sweden（肿瘤病理学系，卡罗林斯卡研究所，索纳）；Department of Clinical Pathology and Cancer Diagnostics, Karolinska University Hospital, Sweden（临床病理学和癌症诊断系，卡罗林斯卡大学医院）；Department of Radiation Physics and Department of Medical and Health Sciences, Linköping University, Sweden（放射物理系和医学与健康科学系，利厄普大学）
PAST: A multimodal single-cell foundation model for histopathology and spatial transcriptomics in cancer
King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Shanghai Academy of Artificial Intelligence for Science（上海科学智能研究院）；Fudan University（复旦大学）；Zhongshan Hospital, Fudan University（复旦大学附属中山医院）
Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis
College of Computer and Information Science, Fujian Agriculture and Forestry University（福建农林大学计算机与信息科学学院）；College of Future Technology, Fujian Agriculture and Forestry University（福建农林大学未来技术学院）；Digital Fujian Institute of Agricultural Big Data, Fujian Agriculture and Forestry University（福建农业大数据数字福建研究院）；Department of Pathology, Clinical Oncology School of Fujian Medical University, and Fujian Cancer Hospital（福建医科大学临床肿瘤学院病理科及福建癌症医院）
Clinically-Informed Modeling for Pediatric Brain Tumor Classification from Whole-Slide Histopathology Images
Dept. of Computer Science（计算机科学系）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；School of Information（信息学院）；Dell Children's Medical Center（德尔儿童医学中心）；Dept. of OREI（OREI部门）；University of Nevada, Reno（内华达大学里诺分校）
Adaptive Multi-Scale Integration Unlocks Robust Cell Annotation in Histopathology Images
Department of Computer and Information Science, University of Pennsylvania（计算机与信息科学系，宾夕法尼亚大学）；Department of Bioengineering, University of Pennsylvania（生物工程系，宾夕法尼亚大学）；Department of Pathology and Laboratory Medicine, University of Pennsylvania（病理学与实验室医学系，宾夕法尼亚大学）；Department of Biostatistics, Epidemiology and Informatics, University of Pennsylvania（生物统计学、流行病学与信息学系，宾夕法尼亚大学）
From Generic to Specialized: A Subspecialty Diagnostic System Powered by Self-Supervised Learning for Cervical Histopathology
Tsinghua University（清华大学）；Ningbo Clinical Pathology Diagnosis Center（宁波市临床病理诊断中心）；University of Oxford（牛津大学）；Research Institute of Tsinghua University in Shenzhen（清华大学深圳研究院）；Greater Bay Area National Center of Technology Innovation（粤港澳大湾区国家技术创新中心）；Zhejiang University School of Medicine（浙江大学医学院）；Hangzhou Adicon Clinical Laboratories Co., Ltd.（杭州艾迪康医学检验中心有限公司）；Wuhan University of Science and Technology（武汉科技大学）；Guangdong Women and Children Hospital（广东省妇幼保健院）；Maternal and Child Health Hospital of Hubei Province（湖北省妇幼保健院）；Shenzhen Shengqiang Technology Co., Ltd.（深圳盛强科技有限公司）；Pengcheng Laboratory（鹏城实验室）；Jinfeng Laboratory（金凤实验室）；The 920th Hospital of the Joint Logistics Support Force of PLA（中国人民解放军联勤保障部队第九二〇医院）；The Integrated Engineering Research Center of Cell Biological Medicine of State and Regions（细胞生物医学国家与区域整合工程研究中心）；Fuzhou University（福州大学）
Slide-based Graph Collaborative Training for Histopathology Whole Slide Image Analysis
University of Science and Technology of China（中国科学技术大学）；the First Affiliated Hospital of USTC（中国科学技术大学第一附属医院）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程高精尖创新中心）
Deep Learning-based Prediction of Breast Cancer Tumor and Immune Phenotypes from Histopathology
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School（哈佛医学院麻省总医院阿蒂努拉·A·马蒂诺斯生物医学成像中心）；Faculty of Engineering, University of Porto（波尔图大学工程学院）；Institute for Systems and Computer Engineering, Technology and Science (INESC TEC)（系统与计算机工程、技术与科学研究院）
LSTMs have a proven track record in analyzing sequential data. But what about unordered instance bags, as found under a Multiple Instance Learning (MIL) setting? While not often used for this, we show LSTMs excell under this setting too. In addition, we show thatLSTMs are capable of indirectly capturing instance-level information us-ing only bag-level annotations. Thus, they can be used to learn instance-level models in a weakly supervised manner. Our empirical evaluation on both simplified (MNIST) and realistic (Lookbook and Histopathology) datasets shows that LSTMs are competitive with or ev…
Symb-xMIL: Symbolic Explanations for Multiple Instance Learning in Digital Pathology
Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）；Machine Learning Group, Technische Universität Berlin（柏林技术大学机器学习组）；Institute of Pathology, Charité Universitätsmedizin（查理研究所病理学部）；Berlin Institute of Health at Charité – Universitätsmedizin Berlin, BIH Biomedical Innovation Academy, BIH Charité Digital Clinician Scientist Program（柏林查理医学研究院健康研究所、BIH生物医学创新学院、BIH查理数字临床科学家项目）；Institute of Pathology, Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学病理学部）；Division of Translational Medical Oncology, DKFZ（转化医学肿瘤学部，德国有机化学研究所）；German Cancer Consortium (DKTK), partner site Munich, a partnership between DKFZ and Ludwig-Maximilians-Universität München (LMU)（德国癌症联盟（DKTK），慕尼黑合作伙伴站点，由DKFZ和路德维希-马克西米利安-慕尼黑大学（LMU）组成）；Department of Artificial Intelligence, Korea University（韩国大学人工智能系）；Max-Planck Institute for Informatics, Saarbrücken, Germany（马克斯·普朗克信息学院，萨尔布吕肯，德国）；Department of Chemistry, Chemical Physics Theory Group, University of Toronto（多伦多大学化学系，化学物理理论组）；Vector Institute for Artificial Intelligence, Toronto, Canada（多伦多人工智能矢量研究所）；Acceleration Consortium, University of Toronto（多伦多大学加速联盟）
