Predicting spatial gene expression from standard H&E images could extend molecular analysis to tissues without spatial transcriptomic measurements. A standard Transformer applies self-attention over spots but ignores where each spot sits in the tissue, whereas a spot's molecular state depends on tissue context at multiple spatial scales. Building on this insight, we introduce MIST, a Multi-context Spatial Transformer that augments the self-attention backbone with two context pathways applied in every layer: a coordinate-invariant local pathway that attends over spatial k-nearest-neighbor spots, and a slide pathway that broadcasts a whole-slide summary to every spot; the model predicts all spots in a single forward pass. Because coordinates enter only through pairwise distances, the local pathway is invariant to translations, rotations, and reflections when image features are fixed. In intra-cohort evaluation across six human organ cohorts from HEST-1k, MIST improves mean Pearson correlation over the strongest evaluated baseline by 9.7%, 11.6%, and 11.4% for 10-, 50-, and 100-gene panels, respectively, with the lowest average MSE and MAE. MIST further improves leave-one-organ-out transfer by 6.0% over the next strongest baseline, attains the best expression-derived domain recovery (highest mean ARI and NMI at every clustering resolution, and higher NMI than the closest baseline on 84% of held-out slides), and reproduces spatial expression patterns of selected cancer-marker genes. Together, these results show that adding local and slide context to a vanilla Transformer backbone is an effective approach to spatial gene expression prediction from histology.
The Hong Kong University of Science and Technology（香港科技大学）
Histology imaging is an essential diagnosis method to finalize the grade and stage of cancer of different tissues, especially for breast cancer diagnosis. Specialists often disagree on the final diagnosis on biopsy tissue due to the complex morphological variety. Although convolutional neural networks (CNN) have advantages in extracting discriminative features in image classification, directly training a CNN on high resolution histology images is computationally infeasible currently. Besides, inconsistent discriminative features often distribute over the whole histology image, which incurs cha…
Integrating Pathology Foundation Models and Spatial Transcriptomics for Cellular Decomposition from Histology Images
School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA（电气与计算机工程学院，佐治亚理工学院）；Department of Biomedical Engineering, Georgia Institute of Technology（生物医学工程系，佐治亚理工学院）；Emory University, Atlanta, GA, USA（埃默里大学）
University of Warwick（华威大学）；The Alan Turing Institute（艾伦·图灵研究所）；University Hospitals Coventry & Warwickshire（考文垂与沃里克郡大学医院）
Convolutional neural networks (CNNs) have been recently used for a variety of histology image analysis. However, availability of a large dataset is a major prerequisite for training a CNN which limits its use by the computational pathology community. In previous studies, CNNs have demonstrated their potential in terms of feature generalizability and transferability accompanied with better performance. Considering these traits of CNN, we propose a simple yet effective method which leverages the strengths of CNN combined with the advantages of including contextual information, particularly desig…
HGP-Mamba: Integrating Histology and Generated Protein Features for Mamba-based Multimodal Survival Risk Prediction
Cancer Hospital of Dalian University of Technology, Shenyang, China（大连理工大学沈阳医院）；School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China（大连理工大学生物医学工程学院）；Key Laboratory of Integrated Circuit and Biomedical Electronic System, Dalian University of Technology, Dalian, China（大连理工大学集成电路与生物医学电子系统重点实验室）
Tissue Image Analytics Centre（组织图像分析中心）；University of Warwick（华威大学）
Semantic segmentation of various tissue and nuclei types in histology images is fundamental to many downstream tasks in the area of computational pathology (CPath). In recent years, Deep Learning (DL) methods have been shown to perform well on segmentation tasks but DL methods generally require a large amount of pixel-wise annotated data. Pixel-wise annotation sometimes requires expert's knowledge and time which is laborious and costly to obtain. In this paper, we present a consistency based semi-supervised learning (SSL) approach that can help mitigate this challenge by exploiting a large amo…
Img2ST-Net: Efficient High-Resolution Spatial Omics Prediction from Whole Slide Histology Images via Fully Convolutional Image-to-Image Learning
Department of Computer Science, Vanderbilt University（计算机科学系，范德比尔特大学）；Weill Cornell Medicine（韦尔·科恩医学中心）；Department of Electrical and Computer Engineering, Vanderbilt University（电气与计算机工程系，范德比尔特大学）；Department of Biostatistics, Vanderbilt University Medical Center（生物统计学系，范德比尔特大学医学中心）；NVIDIA
Shanghai Jiao Tong University（上海交通大学）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；University of New South Wales（新南威尔士大学）；BirenTech Research（壁仞科技研究院）
Stain variations often decrease the generalization ability of deep learning based approaches in digital histopathology analysis. Two separate proposals, namely stain normalization (SN) and stain augmentation (SA), have been spotlighted to reduce the generalization error, where the former alleviates the stain shift across different medical centers using template image and the latter enriches the accessible stain styles by the simulation of more stain variations. However, their applications are bounded by the selection of template images and the construction of unrealistic styles. To address the…
3D-Guided Scalable Flow Matching for Generating Volumetric Tissue Spatial Transcriptomics from Serial Histology
Cambridge Stem Cell Institute, University of Cambridge（剑桥干细胞研究所，剑桥大学）；Cambridge Center for AI in Medicine, University of Cambridge（剑桥人工智能医学中心，剑桥大学）；Department of Medicine, University of Cambridge（剑桥大学医学系）；Computer Science Department, University of Oxford（牛津大学计算机科学系）；Computer Science Department, University of Helsinki（赫尔辛基大学计算机科学系）；Institute for Molecular Medicine Finland, University of Helsinki（芬兰分子医学研究所，赫尔辛基大学）；Wellcome Sanger Institute（沃森桑格研究所）
University of Strasbourg（斯特拉斯堡大学）；INSA-Lyon（法国国立应用科学学院里昂分校）；Université Claude Bernard Lyon 1（克洛德·贝尔纳里昂第一大学）；Hannover Medical School（汉诺威医学院）；Université de Haute Alsace（上阿尔萨斯大学）；University Clinic, Freiburg（弗莱堡大学医院）
Objective: This article presents an automatic image processing framework to extract quantitative high-level information describing the micro-environment of glomeruli in consecutive whole slide images (WSIs) processed with different staining modalities of patients with chronic kidney rejection after kidney transplantation. Methods: This four-step framework consists of: 1) approximate rigid registration, 2) cell and anatomical structure segmentation 3) fusion of information from different stainings using a newly developed registration algorithm 4) feature extraction. Results: Each step of the fr…
Spatially Resolved Gene Expression Prediction from H&E Histology Images via Bi-modal Contrastive Learning
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University Health Network（大学健康网络）；The Donnelly Centre（唐纳利中心）；Toronto General Hospital Research Institute（多伦多总医院研究所）；Canadian Institute for Advanced Research (CIFAR)（加拿大高等研究院（CIFAR））
University of California, Davis（加州大学戴维斯分校）；University of San Francisco Data Institute（旧金山大学数据研究所）；Wicklow AI in Medicine Research Initiative（威克洛医学人工智能研究计划）；University of Texas, San Antonio Health（德克萨斯大学圣安东尼奥健康分校）
MUSE is a novel slide-free imaging technique for histological examination of tissues that can serve as an alternative to traditional histology. In order to bridge the gap between MUSE and traditional histology, we aim to convert MUSE images to resemble authentic hematoxylin- and eosin-stained (H&E) images. We evaluated four models: a non-machine-learning-based color-mapping unmixing-based tool, CycleGAN, DualGAN, and GANILLA. CycleGAN and GANILLA provided visually compelling results that appropriately transferred H&E style and preserved MUSE content. Based on training an automated critic on re…
GC-MoE: Genomics-Guided Cell-Type-Specific Mixture of Experts for Histology-Based Single-Cell Spatial Transcriptomics
Kyushu University（九州大学）；German Research Center for Artificial Intelligence (DFKI GmbH)（德国人工智能研究中心）；RPTU University Kaiserslautern-Landau（科布伦茨-劳恩堡大学）；The University of Osaka（大阪大学）；IntelligentX GmbH；Osaka Metropolitan University（大阪 Metropolitan 大学）
Radboud University Medical Center（拉德堡德大学医学中心）；Canisius-Wilhelmina Hospital（卡尼修斯-威廉明娜医院）；Jeroen Bosch Hospital（耶伦博斯医院）
Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by: (1) noisy and expensive reference standards established by pathologists, (2) lack of generalization due to staining variation across laboratories, and (3) high compu…
Image Data Analytics, Informatics IT, Merck \& Co., Inc., West Point, PA 19486, USA Image Data Analytics, Informatics IT, Merck \& Co., Inc., Rahway, NJ 07065, USA
It is common for pathologists to annotate specific regions of the tissue, such as tumor, directly on the glass slide with markers. Although this practice was helpful prior to the advent of histology whole slide digitization, it often occludes important details which are increasingly relevant to immuno-oncology due to recent advancements in digital pathology imaging techniques. The current work uses a generative adversarial network with cycle loss to remove these annotations while still maintaining the underlying structure of the tissue by solving an image-to-image translation problem. We train…
Vanderbilt University（范德堡大学）；Middle Tennessee State University（中田纳西州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
Diffusion-weighted magnetic resonance imaging (DW-MRI) allows for non-invasive imaging of the local fiber architecture of the human brain at a millimetric scale. Multiple classical approaches have been proposed to detect both single (e.g., tensors) and multiple (e.g., constrained spherical deconvolution, CSD) fiber population orientations per voxel. However, existing techniques generally exhibit low reproducibility across MRI scanners. Herein, we propose a data-driven tech-nique using a neural network design which exploits two categories of data. First, training data were acquired on three squ…
A three in one bottom-up framework for simultaneous semantic segmentation, instance segmentation and classification of multi-organ nuclei in digital cancer histology
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；College of Science & Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）
National University of Sciences and Technology (NUST)（国立科技大学（NUST））
Medical image segmentation assists in computer-aided diagnosis, surgeries, and treatment. Digitize tissue slide images are used to analyze and segment glands, nuclei, and other biomarkers which are further used in computer-aided medical applications. To this end, many researchers developed different neural networks to perform segmentation on histological images, mostly these networks are based on encoder-decoder architecture and also utilize complex attention modules or transformers. However, these networks are less accurate to capture relevant local and global features with accurate boundary …
CHRep: Cross-modal Histology Representation and Post-hoc Calibration for Spatial Gene Expression Prediction
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Key Laboratory of Network System and Network Culture（北京网络系统与网络文化重点实验室）；Peking University Third Hospital（北京大学第三医院）
University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密西根学院）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）
Temperature difference-induced mist adhered to the glass, such as windshield, camera lens, is often inhomogeneous and obscure, easily obstructing the vision and severely degrading the image. Together with adherent raindrops, they bring considerable challenges to various vision systems but without enough attention. Recent methods for other similar problems typically use hand-crafted priors to generate spatial attention maps. In this work, we newly present a problem of image degradation caused by adherent mist and raindrops. An attentive convolutional network is adopted to visually remove the ad…
Microsatellite instability (MSI) is a tumor phenotype whose diagnosis largely impacts patient care in colorectal cancers (CRC), and is associated with response to immunotherapy in all solid tumors. Deep learning models detecting MSI tumors directly from H&E stained slides have shown promise in improving diagnosis of MSI patients. Prior deep learning models for MSI detection have relied on neural networks pretrained on ImageNet dataset, which does not contain any medical image. In this study, we leverage recent advances in self-supervised learning by training neural networks on histology images…
Helmholtz Munich – German Research Center for Environment and Health（亥姆霍兹慕尼黑——德国环境健康研究中心）；Technical University of Munich（慕尼黑工业大学）；Technical University Dresden（德累斯顿工业大学）；University of Leeds（利兹大学）；Maastricht University Medical Center+（马斯特里赫特大学医学中心+）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Johannes Kepler University Hospital Linz（林茨约翰·开普勒大学医院）；Gemeinschaftspraxis Pathologie, Starnberg（施塔恩贝格病理联合诊所）
Background: Deep learning (DL) can extract predictive and prognostic biomarkers from routine pathology slides in colorectal cancer. For example, a DL test for the diagnosis of microsatellite instability (MSI) in CRC has been approved in 2022. Current approaches rely on convolutional neural networks (CNNs). Transformer networks are outperforming CNNs and are replacing them in many applications, but have not been used for biomarker prediction in cancer at a large scale. In addition, most DL approaches have been trained on small patient cohorts, which limits their clinical utility. Methods: In th…
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；Freie Universität Berlin（柏林自由大学）；Technische Hochschule Ingolstadt（英戈尔施塔特工业大学）
Due to morphological similarities, the differentiation of histologic sections of cutaneous tumors into individual subtypes can be challenging. Recently, deep learning-based approaches have proven their potential for supporting pathologists in this regard. However, many of these supervised algorithms require a large amount of annotated data for robust development. We present a publicly available dataset of 350 whole slide images of seven different canine cutaneous tumors complemented by 12,424 polygon annotations for 13 histologic classes, including seven cutaneous tumor subtypes. In inter-rate…
Cross-modal Diffusion Modelling for Super-resolved Spatial Transcriptomics
Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK（剑桥大学应用数学与理论物理系）；School of Science and Engineering, University of Dundee, UK（邓迪大学科学与工程学院）；School of Medicine, University of Dundee, UK（邓迪大学医学院）；Zhejiang Lab, China（浙江实验室）
High-Resolution Spatial Transcriptomics from Histology Images using HisToSGE
School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；College of Nursing Health Sciences, Yunnan Open University（云南开放大学护理与健康科学学院）
University of Warwick（华威大学）；Tissue Image Analytics (TIA) Centre, University of Warwick（华威大学组织图像分析中心）
The problem of recognizing various types of tissues present in multi-gigapixel histology images is an important fundamental pre-requisite for downstream analysis of the tumor microenvironment in a bottom-up analysis paradigm for computational pathology. In this paper, we propose a deep dictionary learning approach to solve the problem of tissue phenotyping in histology images. We propose deep Multi-Resolution Dictionary Learning (deepMRDL) in order to benefit from deep texture descriptors at multiple different spatial resolutions. We show the efficacy of the proposed approach through extensive…
Disentangled Multi-modal Learning of Histology and Transcriptomics for Cancer Characterization
Department of Clinical Neurosciences, University of Cambridge, UK（剑桥大学临床神经科学系）；Department of Health Technology & Informatics, The Hong Kong Polytechnic University（香港理工大学健康科技与信息学系）；Department of Statistics and Actuarial Science, The University of Hong Kong（香港大学统计与精算科学系）；Department of Clinical Neurosciences and Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学临床神经科学系和应用数学与理论物理系；邓迪大学科学与工程学院和医学院）；School of Science and Engineering and School of Medicine, University of Dundee, UK
Multi-Scale Hybrid Vision Transformer for Learning Gastric Histology: AI-Based Decision Support System for Gastric Cancer Treatment
KAIST（韩国科学技术院）；Chungnam National University School of Medicine（忠南大学医学院）；Chungnam National University Hospital（忠南大学医院）；Chungnam National University Sejong Hospital（忠南大学世宗医院）
本文提出基于两阶段混合 ViT 与多尺度自注意力的 AI 系统，实现胃癌病理五分类并匹配治疗指南，在多中心切片中验证可提升灵敏度并缩短筛查时间。
