Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Stony Brook University（石溪大学）；Shaukat Khanum Cancer Hospital（肖卡特·卡努姆癌症医院）；Australian National University（澳大利亚国立大学）
Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations. In this paper, we explore how the combination of domain specific natural language information with such hierarchical visual representations can benefit rich representation learning for medical image tasks. Building on automated language description generation for features visible in histopathology images, we present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. We explore contrastive objectives and granular language description based text alignment at multiple hierarchies to inject language modality information into the visual representations. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets. Our framework also provides better interpretability with our language aligned representation space. Code is available at https://github.com/Hasindri/HLSS.
Hierarchical Classification for Improved Histopathology Image Analysis
School of Electrical Engineering, Korea University, Seoul, Korea（韩国大学电气工程学院）；Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul, Korea（韩国天主大学医学院医院病理系）
Northeastern University（东北大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Stevens Institute of Technology（史蒂文斯理工学院）；China Medical University（中国医科大学）；Liaoning Cancer Hospital and Institute（辽宁省肿瘤医院暨研究所）；Chengdu University of Information Technology（成都信息工程大学）
In the Gastric Histopathology Image Classification (GHIC) tasks, which are usually weakly supervised learning missions, there is inevitably redundant information in the images. Therefore, designing networks that can focus on effective distinguishing features has become a popular research topic. In this paper, to accomplish the tasks of GHIC superiorly and to assist pathologists in clinical diagnosis, an intelligent Hierarchical Conditional Random Field based Attention Mechanism (HCRF-AM) model is proposed. The HCRF-AM model consists of an Attention Mechanism (AM) module and an Image Classifica…
Hierarchical Self-Supervised Adversarial Training for Robust Vision Models in Histopathology
Mohamed Bin Zayed University of Artificial Intelligence（Mohamed Bin Zayed人工智能大学）；Khalifa University（卡勒比大学）；Linköping University（林霍尔姆大学）；Australian National University（澳大利亚国立大学）
NYU Grossman School of Medicine（纽约大学格罗斯曼医学院）；University of Toronto（多伦多大学）；Microsoft Research（微软研究院）
Scarcity of labeled histopathology data limits the applicability of deep learning methods to under-profiled cancer types and labels. Transfer learning allows researchers to overcome the limitations of small datasets by pre-training machine learning models on larger datasets similar to the small target dataset. However, similarity between datasets is often determined heuristically. In this paper, we propose a principled notion of distance between histopathology datasets based on a hierarchical generalization of optimal transport distances. Our method does not require any training, is agnostic t…
Weakly Supervised Spatial Grounding for Discriminative Attention-Based Ultrasound-Histopathology Alignment in Prostate Cancer Grading
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Ordensklinikum Linz（林茨教团医院）；University of Alberta（阿尔伯塔大学）；Exact Imaging（Exact Imaging公司）
Hierarchical Prototype-based Domain Priors for Multiple Instance Learning in Multimodal Histopathology Analysis
College of Computer and Information Science, Fujian Agriculture and Forestry University（福建农林大学计算机与信息科学学院）；College of Future Technology, Fujian Agriculture and Forestry University（福建农林大学未来技术学院）；Digital Fujian Institute of Agricultural Big Data, Fujian Agriculture and Forestry University（福建农业大数据数字福建研究院）；Department of Pathology, Clinical Oncology School of Fujian Medical University, and Fujian Cancer Hospital（福建医科大学临床肿瘤学院病理科及福建癌症医院）
Single-Stage Hierarchical Rectification for Weakly Supervised Histopathology Segmentation
VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam（越南河内VinUniversity VinUni-Illinois智慧健康中心）；The Computer Vision and Medical AI Lab, VinUniversity, Hanoi, Vietnam（越南河内VinUniversity计算机视觉与医学人工智能实验室）；Posts and Telecommunications Institute of Technology, Hanoi, Vietnam（越南河内邮电技术学院）
Beihang University（北京航空航天大学）；Hefei University of Technology（合肥工业大学）
Transformer has been widely used in histopathology whole slide image (WSI) classification for the purpose of tumor grading, prognosis analysis, etc. However, the design of token-wise self-attention and positional embedding strategy in the common Transformer limits the effectiveness and efficiency in the application to gigapixel histopathology images. In this paper, we propose a kernel attention Transformer (KAT) for histopathology WSI classification. The information transmission of the tokens is achieved by cross-attention between the tokens and a set of kernels related to a set of positional …
Advancement in digital pathology and artificial intelligence has enabled deep learning-based computer vision techniques for automated disease diagnosis and prognosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized, making them infeasible to be used directly for training deep neural networks. Hence, for modeling, a two-stage approach is adopted: Patch representations are extracted first, followed by the aggregation for WSI prediction. These approaches require detailed pixel-level annotations for training the patch encoder. However, obtaining thes…
SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome
Wellcome Sanger Institute（沃森桑格研究所）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；Institute of AI for Health（人工智能与健康研究所）；Cambridge Stem Cell Institute（剑桥干细胞研究所）
From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformer
University of Science and Technology Beijing（北京科技大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；Changhai Hospital（长海医院）；Peking University（北京大学）
RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistency
Stony Brook University（石英溪大学）；Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital and Harvard Medical School（阿提诺拉A.马丁努斯生物医学影像中心，麻省总医院和哈佛医学院）；Department of Biomedical Data Science, Stanford University（生物医学数据科学系，斯坦福大学）；Department of Pathology and Cell Biology, Columbia University（病理学与细胞生物学系，哥伦比亚大学）
Hispathological image segmentation algorithms play a critical role in computer aided diagnosis technology. The development of weakly supervised segmentation algorithm alleviates the problem of medical image annotation that it is time-consuming and labor-intensive. As a subset of weakly supervised learning, Multiple Instance Learning (MIL) has been proven to be effective in segmentation. However, there is a lack of related information between instances in MIL, which limits the further improvement of segmentation performance. In this paper, we propose a novel weakly supervised method for pixel-l…
MATCH: Multi-faceted Adaptive Topo-Consistency for Semi-Supervised Histopathology Segmentation
Stony Brook University（斯通布罗克大学）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）；Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系）
ConStruct: Structural Distillation of Foundation Models for Prototype-Based Weakly Supervised Histopathology Segmentation
Ho Chi Minh City University of Technology（胡志明市技术大学）；University of Technology Sydney（悉尼技术大学）；University of Houston（休斯顿大学）；University of Information Technology（信息科技大学）；College of Medical Science and Technology, Taipei Medical University（台北医学院医学科技学院）；Department of Computer Science, Emory University（埃默里大学计算机科学系）；Montefiore Medical Center, Albert Einstein College of Medicine（蒙特福伊医疗中心，阿尔伯特·爱因斯坦医学院）；School of Computer Science, Carnegie Mellon University（卡内基梅隆大学计算机科学学院）
Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading
The University of Hong Kong（香港大学）；City University of Hong Kong（香港城市大学）；The Hang Seng University of Hong Kong（香港恒生大学）；Stanford University（斯坦福大学）；Centre for Artificial Intelligence and Robotics (CAIR), HKISI-CAS（中国科学院香港人工智能与机器人中心）
Joint analysis of multiple biomarker images and tissue morphology is important for disease diagnosis, treatment planning and drug development. It requires cross-staining comparison among Whole Slide Images (WSIs) of immuno-histochemical and hematoxylin and eosin (H&E) microscopic slides. However, automatic, and fast cross-staining alignment of enormous gigapixel WSIs at single-cell precision is challenging. In addition to morphological deformations introduced during slide preparation, there are large variations in cell appearance and tissue morphology across different staining. In this paper, …
Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading
Dartmouth College（达特茅斯学院）；Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Massachusetts General Hospital（麻省总医院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希契科克医疗中心）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）
MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention
The University of Sydney（悉尼大学）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（集成空天地海大数据应用技术国家工程实验室）；School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机科学与工程学院）；University of Maryland（马里兰大学）；Ningbo Institute of Northwestern Polytechnical University（西北工业大学宁波学院）
Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University（北京航空航天大学生物医学工程北京高级创新中心）；School of Software, Hefei University of Technology（合肥工业大学软件学院）；Tianjin Institute of Industrial Biotechnology, Chinese Academy of Sciences（中国科学院天津工业生物技术研究所）
Content-based histopathological image retrieval (CBHIR) has become popular in recent years in the domain of histopathological image analysis. CBHIR systems provide auxiliary diagnosis information for pathologists by searching for and returning regions that are contently similar to the region of interest (ROI) from a pre-established database. While, it is challenging and yet significant in clinical applications to retrieve diagnostically relevant regions from a database that consists of histopathological whole slide images (WSIs) for a query ROI. In this paper, we propose a novel framework for …
Variously stained histology slices are routinely used by pathologists to assess extracted tissue samples from various anatomical sites and determine the presence or extent of a disease. Evaluation of sequential slides is expected to enable a better understanding of the spatial arrangement and growth patterns of cells and vessels. In this paper we present a practical two-step approach based on diffeomorphic registration to align digitized sequential histopathology stained slides to each other, starting with an initial affine step followed by the estimation of a detailed deformation field.
Retrieval-Guided Generation for Safer Histopathology Image Captioning
Kimia Lab, Department of Artificial Intelligence \& Informatics, Mayo Clinic, Rochester, MN, USA；Department of Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock, AR, USA；Lane Department of Computer Science；Electrical Engineering, West Virginia University, Morgantown, WV, USA；Department of Computer Science；Engineering, Princeton University, Princeton, NJ, USA；Department of Computer Sciences, University of Wisconsin--Madison, Madison, WI, USA
GRAPHITE: Graph-Based Interpretable Tissue Examination for Enhanced Explainability in Breast Cancer Histopathology
School of Computer Science and Engineering（计算机科学与工程学院）；University of New South Wales（新南威尔士大学）；St. George and Sutherland Clinical School（圣乔治与萨瑟兰临床学院）；Department of Anatomical Pathology（解剖病理学部）；NSW Health Pathology（新南威尔士州卫生病理学）；Cancer Care Centre（癌症护理中心）
Pathologist Attention-Aligned Report Generation for Prostate Histopathology
Department of Computer Science, Stony Brook University（纽约州立大学石溪分校计算机科学系）；Department of Biomedical Informatics, Stony Brook University（纽约州立大学石溪分校生物医学信息学系）；Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit（巴黎萨克雷大学、中央理工高等电力学院、古斯塔夫·鲁西研究所、法国国家健康与医学研究院、PRISM综合大学医院、癌症数据科学单元）；Université Paris-Saclay, CentraleSupélec, MICS Laboratory（巴黎萨克雷大学、中央理工高等电力学院、MICS实验室）；Department of Pathology and Laboratory Medicine, Northwell Health Laboratories（诺斯韦尔健康实验室病理与检验医学部）；Department of Pathology, University of Utah School of Medicine（犹他大学医学院病理系）；Department of Psychology, Stony Brook University（纽约州立大学石溪分校心理学系）
Pediatric brain tumor classification using digital histopathology and deep learning: evaluation of SOTA methods on a multi-center Swedish cohort
Department of Biomedical Engineering, Linköping University, Sweden（生物医学工程系，利厄普大学）；Center for Medical Image Science and Visualization, Linköping University, Sweden（医学影像科学与可视化中心，利厄普大学）；Crown Princess Victoria Children’s Hospital and Department of Health, Medicine and Caring Sciences, Linköping University, Sweden（维多利亚公主儿童医院和健康、医学与关怀科学系，利厄普大学）；Department of Radiology and Department of Health, Medicine and Caring Sciences, Linköping University, Sweden（放射科和健康、医学与关怀科学系，利厄普大学）；Department of Oncology-Pathology, Karolinska Institutet, Solna, Sweden（肿瘤病理学系，卡罗林斯卡研究所，索纳）；Department of Clinical Pathology and Cancer Diagnostics, Karolinska University Hospital, Sweden（临床病理学和癌症诊断系，卡罗林斯卡大学医院）；Department of Radiation Physics and Department of Medical and Health Sciences, Linköping University, Sweden（放射物理系和医学与健康科学系，利厄普大学）
RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation
Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李科钦医学院）；MedVisAI Lab（医学影像人工智能实验室）；Centre of AI in Medicine, Singapore（新加坡人工智能医学中心）；Chinese University of Hong Kong, Hong Kong SAR, China（香港中文大学（深圳））；Department of Computer Science and Engineering（计算机科学与工程系）
A Hierarchical Transformer Encoder to Improve Entire Neoplasm Segmentation on Whole Slide Image of Hepatocellular Carcinoma
Université Paris Cité（巴黎西岱大学）；University of Applied Sciences of Western Switzerland (HES-SO Valais)（瑞士西部应用科学大学（瓦莱州高等专业学院））；French University Institute (IUF)（法国大学研究院）
针对肝细胞癌 WSI 全肿瘤分割难题，提出分层 Transformer 架构 HiTrans，在 4096×4096 图像块上学习全局依赖并提升分割性能。
Recently, medical report generation, which aims to automatically generate a long and coherent descriptive paragraph of a given medical image, has received growing research interests. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias: the normal visual regions dominate the dataset over the abnormal visual regions, and 2) the very long sequence. To alleviate above two problems, we propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) …
Northeastern University（东北大学）；MBIE College（MBIE学院）；Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education（教育部医学影像与智能分析工程研究中心）；Microscopic Image and Medical Image Analysis Group（显微图像与医学图像分析组）
For the Convolutional Neural Networks (CNNs) applied in the intelligent diagnosis of gastric cancer, existing methods mostly focus on individual characteristics or network frameworks without a policy to depict the integral information. Mainly, Conditional Random Field (CRF), an efficient and stable algorithm for analyzing images containing complicated contents, can characterize spatial relation in images. In this paper, a novel Hierarchical Conditional Random Field (HCRF) based Gastric Histopathology Image Segmentation (GHIS) method is proposed, which can automatically localize abnormal (cance…
Institute of AI for Health, Helmholtz Munich, Germany International School of Medicine, Istanbul Medipol University, T\"urkiye Munich Leukemia Laboratory, Germany Department of Medicine III, Ludwig-Maximilian-University Hospital, Germany Department of Physics, University of Munich, Germany Munich Center for Machine Learning (MCML), Germany DKTK, German Cancer Consortium, Germany
Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation
Xinjiang University（新疆大学）；Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center（新疆多模态智能处理与信息安全工程技术研究中心）；Tsinghua University（清华大学）
Radboud University Medical Center（拉德堡德大学医学中心）；Institute of Experimental Immunology, University of Zurich（苏黎世大学实验免疫学研究所）；Center for Medical Image Science and Visualization, Linköping University（林雪平大学医学影像科学与可视化中心）
We propose HookNet, a semantic segmentation model for histopathology whole-slide images, which combines context and details via multiple branches of encoder-decoder convolutional neural networks. Concentricpatches at multiple resolutions with different fields of view are used to feed different branches of HookNet, and intermediate representations are combined via a hooking mechanism. We describe a framework to design and train HookNet for achieving high-resolution semantic segmentation and introduce constraints to guarantee pixel-wise alignment in feature maps during hooking. We show the advan…
