Leveraging image captions for selective whole slide image annotation
FAU Erlangen-Nürnberg（埃尔朗根-纽伦堡大学）；Technische Hochschule Ingolstadt（英戈尔施塔特应用技术大学）；Flensburg University of Applied Sciences（弗伦斯堡应用技术大学）；Universität Würzburg（维尔茨堡大学）
Jingna Qiu, Marc Aubreville, Frauke Wilm, Mathias Öttl, Jonas Utz, Maja Schlereth, Katharina Breininger
Acquiring annotations for whole slide images (WSIs)-based deep learning tasks, such as creating tissue segmentation masks or detecting mitotic figures, is a laborious process due to the extensive image size and the significant manual work involved in the annotation. This paper focuses on identifying and annotating specific image regions that optimize model training, given a limited annotation budget. While random sampling helps capture data variance by collecting annotation regions throughout the WSIs, insufficient data curation may result in an inadequate representation of minority classes. Recent studies proposed diversity sampling to select a set of regions that maximally represent unique characteristics of the WSIs. This is done by pretraining on unlabeled data through self-supervised learning and then clustering all regions in the latent space. However, establishing the optimal number of clusters can be difficult and not all clusters are task-relevant. This paper presents prototype sampling, a new method for annotation region selection. It discovers regions exhibiting typical characteristics of each task-specific class. The process entails recognizing class prototypes from extensive histopathology image-caption databases and detecting unlabeled image regions that resemble these prototypes. Our results show that prototype sampling is more effective than random and diversity sampling in identifying annotation regions with valuable training information, resulting in improved model performance in semantic segmentation and mitotic figure detection tasks. Code is available at https://github.com/DeepMicroscopy/Prototype-sampling.
Open-vocabulary instance segmentation aims at segmenting novel classes without mask annotations. It is an important step toward reducing laborious human supervision. Most existing works first pretrain a model on captioned images covering many novel classes and then finetune it on limited base classes with mask annotations. However, the high-level textual information learned from caption pretraining alone cannot effectively encode the details required for pixel-wise segmentation. To address this, we propose a cross-modal pseudo-labeling framework, which generates training pseudo masks by aligni…
The University of Queensland（昆士兰大学）；Sullivan Nicolaides Pathology（沙利文尼古拉德斯病理学公司）
The difficulty of processing gigapixel whole slide images (WSIs) in clinical microscopy has been a long-standing barrier to implementing computer aided diagnostic systems. Since modern computing resources are unable to perform computations at this extremely large scale, current state of the art methods utilize patch-based processing to preserve the resolution of WSIs. However, these methods are often resource intensive and make significant compromises on processing time. In this paper, we demonstrate that conventional patch-based processing is redundant for certain WSI classification tasks whe…
Computational histopathology has made significant strides in the past few years, slowly getting closer to clinical adoption. One area of benefit would be the automatic generation of diagnostic reports from H\&E-stained whole slide images which would further increase the efficiency of the pathologists' routine diagnostic workflows. In this study, we compiled a dataset (PatchGastricADC22) of histopathological captions of stomach adenocarcinoma endoscopic biopsy specimens, which we extracted from diagnostic reports and paired with patches extracted from the associated whole slide images. The data…
GNN-ViTCap: GNN-Enhanced Multiple Instance Learning with Vision Transformers for Whole Slide Image Classification and Captioning
Centre for Pattern Analysis and Machine Intelligence, Department of Electrical and Computer Engineering, University of Waterloo（模式分析与机器智能中心，电气与计算机工程系，滑铁库大学）；Machine Learning Department at Mohamed bin Zayed University of Artificial Intelligence（人工智能穆罕默德·本·扎耶德大学机器学习系）
Most image captioning frameworks generate captions directly from images, learning a mapping from visual features to natural language. However, editing existing captions can be easier than generating new ones from scratch. Intuitively, when editing captions, a model is not required to learn information that is already present in the caption (i.e. sentence structure), enabling it to focus on fixing details (e.g. replacing repetitive words). This paper proposes a novel approach to image captioning based on iterative adaptive refinement of an existing caption. Specifically, our caption-editing mod…
Department of Computer Science and Engineering, University of Minnesota（明尼苏达大学计算机科学与工程系）
Visual attention has shown usefulness in image captioning, with the goal of enabling a caption model to selectively focus on regions of interest. Existing models typically rely on top-down language information and learn attention implicitly by optimizing the captioning objectives. While somewhat effective, the learned top-down attention can fail to focus on correct regions of interest without direct supervision of attention. Inspired by the human visual system which is driven by not only the task-specific top-down signals but also the visual stimuli, we in this work propose to use both types o…
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）
Local representation learning has been a key challenge to promote the performance of the histopathological whole slide images analysis. The previous representation learning methods followed the supervised learning paradigm. However, manual annotation for large-scale WSIs is time-consuming and labor-intensive. Hence, the self-supervised contrastive learning has recently attracted intensive attention. The present contrastive learning methods treat each sample as a single class, which suffers from class collision problems, especially in the domain of histopathology image analysis. In this paper, …
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Image captioning is one of the most challenging tasks in AI, which aims to automatically generate textual sentences for an image. Recent methods for image captioning follow encoder-decoder framework that transforms the sequence of salient regions in an image into natural language descriptions. However, these models usually lack the comprehensive understanding of the contextual interactions reflected on various visual relationships between objects. In this paper, we explore explicit and implicit visual relationships to enrich region-level representations for image captioning. Explicitly, we bui…
Hepato-LLaVA: An Expert MLLM with Sparse Topo-Pack Attention for Hepatocellular Pathology Analysis on Whole Slide Images
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）；Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（pathology department, 国家癌症中心/国家癌症临床研究中心/癌症医院, 中国医学科学院和北京协和医学院）
Queen Mary University of London（伦敦玛丽女王大学）；Karolinska University Hospital（卡罗林斯卡大学医院）
Histopathological cancer diagnosis is based on visual examination of stained tissue slides. Hematoxylin and eosin (H\&E) is a standard stain routinely employed worldwide. It is easy to acquire and cost effective, but cells and tissue components show low-contrast with varying tones of dark blue and pink, which makes difficult visual assessments, digital image analysis, and quantifications. These limitations can be overcome by IHC staining of target proteins of the tissue slide. IHC provides a selective, high-contrast imaging of cells and tissue components, but their use is largely limited by a …
PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology（哈尔滨工业大学（深圳）计算机科学与技术学院）；National University of Singapore, Department of Electronic and Computer Engineering（新加坡国立大学电子与计算机工程系）
KHiM-Mamba: Injecting Pathology Knowledge into Mamba via Hidden-State Modulation for Whole Slide Image Analysis
The Hong Kong University of Science and Technology（香港科技大学）；Chongqing University Cancer Hospital（重庆大学附属肿瘤医院）；Chongqing University School of Medicine（重庆大学医学院）
2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification
Stony Brook University（石溪大学）；Korea University（韩国大学）；Concordia University（康科迪亚大学）；Mila–Quebec AI Institute（魁北克AI研究院）；University of Montreal Hospital Center（蒙特利尔医院中心）
CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology
Westlake University（西湖大学）；Zhejiang University（浙江大学）；Harvard University（哈佛大学）；The Ohio State University（俄亥俄州立大学）；University of the Chinese Academy of Sciences（中国科学院大学）
QCAgent: An agentic framework for quality-controllable pathology report generation from whole slide image
School of Biomedical Engineering, Division of Life Sciences；Medicine, University of Science；Technology of China, Hefei, Anhui, 230026, P.R. China Center for Medical Imaging, Robotics, Analytic Computing \& Learning (MIRACLE), Suzhou Institute for Advance Research, USTC, 215123, P.R. China Chinese PLA General Hospital Ninth Medical Center, China Hong Kong University of Science
Initialization matters in few-shot adaptation of vision-language models for histopathological image classification
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano (HUMAN-Tech)（人类中心技术研究所）；Universitat Politécnica de Valencia（巴塞罗那理工大学）；Valencian Graduate School and Research Network for Artificial Intelligence (valgrAI)（瓦伦西亚人工智能研究生院和研究网络）
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
Department of Computer Science, Khalifa University of Science and Technology（卡利法科技大学计算机科学系）；Information Technology University（信息技术大学）；KAU（卡乌大学）；University of the Western Australia（西澳大学）
Advancing Multiple Instance Learning with Continual Learning for Whole Slide Imaging
Dept. of Computer Science（计算机科学系）；City University of Hong Kong（香港城市大学）；Noah’s Ark Lab, Huawei Canada（华为加拿大诺亚实验室）；Guangzhou Bingli Technology Co., Ltd.（广州炳利科技有限公司）
A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model
The Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Guangdong Provincial Key Laboratory of Molecular Tumor Pathology（广东省分子肿瘤病理学重点实验室）；Jinfeng Laboratory（金凤实验室）；Shandong First Medical University（山东第一医科大学）；Shandong Provincial Qianfoshan Hospital（山东省千佛山医院）；The Third Affiliated Hospital of Kunming Medical University（昆明医科大学第三附属医院）；Yunnan Cancer Hospital（云南省肿瘤医院）；The Chinese University of Hong Kong（香港中文大学）；Prince of Wales Hospital（威尔士亲王医院）；Sun Yat-sen University（中山大学）
MMNavAgent: Multi-Magnification WSI Navigation Agent for Clinically Consistent Whole-Slide Analysis
Institute of Pathology, Technical University of Munich, Germany（慕尼黑技术大学病理研究所）；Munich Data Science Institute (MDSI), Munich, Germany（慕尼黑数据科学研究所）；Munich Center for Machine Learning (MCML), Munich, Germany（慕尼黑机器学习中心）；Computer Aided Medical Procedures (CAMP), TU Munich, Munich, Germany（计算机辅助医疗程序（CAMP），慕尼黑技术大学）；Dalian University of Technology（大连理工大学）；Cancer Hospital of Dalian University of Technology, Shenyang（大连理工大学肿瘤医院）；Department of Human Pathology, Juntendo University Graduate School of Medicine（立命大学医学研究生院人类病理部门）；University of Science and Technology of China（中国科学技术大学）；Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医学机器人研究所）；Northwest University of China（中国西北大学）
The Pennsylvania State University（宾夕法尼亚州立大学）；Lister Hill National Center for Biomedical Communications（利斯特希尔国家生物医学通信中心）；National Library of Medicine（国家医学图书馆）；Penn State Health Milton S. Hershey Medical Center（宾夕法尼亚州立大学健康米尔顿·S·赫尔希医学中心）；Penn State College Of Medicine（宾夕法尼亚州立大学医学院）
Histopathological analysis is the present gold standard for precancerous lesion diagnosis. The goal of automated histopathological classification from digital images requires supervised training, which requires a large number of expert annotations that can be expensive and time-consuming to collect. Meanwhile, accurate classification of image patches cropped from whole-slide images is essential for standard sliding window based histopathology slide classification methods. To mitigate these issues, we propose a carefully designed conditional GAN model, namely HistoGAN, for synthesizing realisti…
Petuum, Inc.（Petuum公司）；Cornell University（康奈尔大学）；UiT The Arctic University of Norway（挪威北极大学）；Carnegie Mellon University（卡内基梅隆大学）
Convolutional neural networks have led to significant breakthroughs in the domain of medical image analysis. However, the task of breast cancer segmentation in whole-slide images (WSIs) is still underexplored. WSIs are large histopathological images with extremely high resolution. Constrained by the hardware and field of view, using high-magnification patches can slow down the inference process and using low-magnification patches can cause the loss of information. In this paper, we aim to achieve two seemingly conflicting goals for breast cancer segmentation: accurate and fast prediction. We p…
MambaMIL+: Modeling Long-Term Contextual Patterns for Gigapixel Whole Slide Image
Department of Computer Science and Engineering, Hong Kong University of Science and Technology（计算机科学与工程系，香港科学与技术大学）；Department of Pathology, Nanfang Hospital, School of Basic Medical Sciences, Southern Medical University（病理学系，南方医科大学）；th Hospital of Joint Logistic Support Force, PLA（联合后勤保障部队900医院）；Department of Pathology, Zhujiang Hospital, Southern Medical University（病理学系，南方医科大学）；Department of Computer Science and Engineering, the Department of Chemical and Biological Engineering, the Division of Life Science, the State Key Laboratory of Nervous System Disorders, Hong Kong University of Science and Technology（计算机科学与工程系、化学与生物工程系、生命科学 division、神经系统紊乱国家重点实验室，香港科学与技术大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深圳-香港协同创新研究院）
Queen Mary University of London（伦敦玛丽女王大学）；Communication University of Zhejiang（浙江传媒学院）；University of Leicester（莱斯特大学）
Cancer segmentation in whole-slide images is a fundamental step for viable tumour burden estimation, which is of great value for cancer assessment. However, factors like vague boundaries or small regions dissociated from viable tumour areas make it a challenging task. Considering the usefulness of multi-scale features in various vision-related tasks, we present a structure-aware scale-adaptive feature selection method for efficient and accurate cancer segmentation. Based on a segmentation network with a popular encoder-decoder architecture, a scale-adaptive module is proposed for selecting mor…
Multimodal Whole Slide Foundation Model for Pathology
Mass General Brigham, Harvard Medical School（哈佛医学院，麻省总医院布里格姆医疗系统）；Dana-Farber Cancer Institute（丹娜法伯癌症研究院）；Broad Institute of Harvard and MIT（哈佛与麻省理工学院博德研究所）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑中心——德国环境与健康研究中心）；School of Computation, Information and Technology, Technical University of Munich（慕尼黑工业大学计算、信息与技术学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology（麻省理工学院电气工程与计算机科学系）；Harvard-MIT Division of Health Sciences and Technology, Massachusetts Institute of Technology（麻省理工学院哈佛-麻省卫生科学与技术分部）；Department of Pathology, Pusan National University（釜山大学病理学系）；Department of Pathology and Laboratory Medicine, Emory University School of Medicine（埃默里大学医学院病理学与检验医学系）；Harvard Data Science Initiative, Harvard University（哈佛大学哈佛数据科学计划）；Department of Systems Biology, Harvard Medical School（哈佛医学院系统生物学系）；Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo（东京大学医学部预防医学系）
A deep learning framework for efficient pathology image analysis
TUD Dresden University of Technology（德累斯顿工业大学）；University Medical Center Mainz（美因茨大学医学中心）；National Center for Tumor Diseases Dresden (NCT/UCC)（德累斯顿国家肿瘤疾病中心）；University Hospital RWTH Aachen（亚琛工业大学附属医院）；Johannes Kepler University Linz（约翰内斯·开普勒大学林茨分校）；University of Bern（伯尔尼大学）；Mayo Clinic（妙佑医疗国际）；Medical University of Vienna（维也纳医科大学）；International Agency for Research on Cancer, World Health Organization（世界卫生组织国际癌症研究机构）；Imperial College London（帝国理工学院）；Fred Hutchinson Cancer Center（弗雷德·哈钦森癌症中心）；University Hospital Schleswig-Holstein（石勒苏益格-荷尔斯泰因大学医院）；University of Washington（华盛顿大学）；European Institute of Oncology IRCCS（欧洲肿瘤学研究所IRCCS）；University of Milan（米兰大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；German Cancer Consortium (DKTK)（德国癌症联盟）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）
TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany（马克斯·普朗克研究所信息学院，萨尔兰信息学院，德国萨尔布吕肯）；Department of Language Science and Technology, Saarland University, Saarbrücken, Germany（语言科学与技术系，萨尔兰大学，德国萨尔布吕肯）
CRISP -- Clustering-Based Redundancy-Reduced Instance Sampling for Pathology Case Representation and Retrieval
Kimia Lab, Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, USA（Kimia实验室，人工智能与信息学系，梅奥诊所，罗切斯特，明尼苏达州，美国）；DICE Lab, Department of Electrical and Computer Engineering, University of Illinois Chicago, IL, USA（DICE实验室，电气与计算机工程系，伊利诺伊大学芝加哥分校，伊利诺伊州，美国）；MD Kimia Lab, Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, USA（MD Kimia实验室，人工智能与信息学系，梅奥诊所，罗切斯特，明尼苏达州，美国）；PhD Kimia Lab, Department of Artificial Intelligence & Informatics, Mayo Clinic, Rochester, MN, USA（PhD Kimia实验室，人工智能与信息学系，梅奥诊所，罗切斯特，明尼苏达州，美国）；Division of Computational Pathology and Informatics, Mayo Clinic, Rochester, MN, USA（计算病理学与信息学部，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, USA（实验室医学与病理学系，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Breast and Melanoma Surgical Oncology, Comprehensive Cancer Center, Mayo Clinic, Rochester, MN, USA（乳腺和黑色素瘤外科肿瘤学系，综合癌症中心，梅奥诊所，罗切斯特，明尼苏达州，美国）；Department of Oncology, Comprehensive Cancer Center, Mayo Clinic, Rochester, MN, USA（肿瘤学系，综合癌症中心，梅奥诊所，罗切斯特，明尼苏达州，美国）；PhD H.R. Tizhoosh
