A key step in understanding the spatial organization of cells and tissues is the ability to construct generative models that accurately reflect that organization. In this paper, we focus on building generative models of electron microscope (EM) images in which the positions of cell membranes and mitochondria have been densely annotated, and propose a two-stage procedure that produces realistic images using Generative Adversarial Networks (or GANs) in a supervised way. In the first stage, we synthesize a label "image" given a noise "image" as input, which then provides supervision for EM image synthesis in the second stage. The full model naturally generates label-image pairs. We show that accurate synthetic EM images are produced using assessment via (1) shape features and global statistics, (2) segmentation accuracies, and (3) user studies. We also demonstrate further improvements by enforcing a reconstruction loss on intermediate synthetic labels and thus unifying the two stages into one single end-to-end framework.
GANs vs. Diffusion Models for virtual staining with the HER2match dataset
The Netherlands Cancer Institute, Amsterdam, The Netherlands（荷兰癌症研究所）；Faculty of Engineering, University of Porto, Porto, Portugal（葡萄牙波尔图大学工程学院）；IMP Diagnostics, Porto, Portugal（IMP诊断（葡萄牙波尔图））
Robert Bosch Engineering and Business Solutions Private Limited（罗伯特·博世工程与商业解决方案私人有限公司）；HEC Paris（巴黎高等商学院）
Generative Adversarial Network (GAN) is a current focal point of research. The body of knowledge is fragmented, leading to a trial-error method while selecting an appropriate GAN for a given scenario. We provide a comprehensive summary of the evolution of GANs starting from its inception addressing issues like mode collapse, vanishing gradient, unstable training and non-convergence. We also provide a comparison of various GANs from the application point of view, its behaviour and implementation details. We propose a novel framework to identify candidate GANs for a specific use case based on ar…
University of Tübingen（图宾根大学）；University Hospital and Comprehensive Cancer Center Tübingen（图宾根大学医院暨综合癌症中心）
Understanding the interactions of different cell types inside the immune tumor microenvironment (iTME) is crucial for the development of immunotherapy treatments as well as for predicting their outcomes. Highly multiplexed tissue imaging (HMTI) technologies offer a tool which can capture cell properties of tissue samples by measuring expression of various proteins and storing them in separate image channels. HMTI technologies can be used to gain insights into the iTME and in particular how the iTME differs for different patient outcome groups of interest (e.g., treatment responders vs. non-res…
It is increasingly common in many types of natural and physical systems (especially biological systems) to have different types of measurements performed on the same underlying system. In such settings, it is important to align the manifolds arising from each measurement in order to integrate such data and gain an improved picture of the system. We tackle this problem using generative adversarial networks (GANs). Recently, GANs have been utilized to try to find correspondences between sets of samples. However, these GANs are not explicitly designed for proper alignment of manifolds. We present…
Kheiron Medical Technologies Ltd.（Kheiron医疗技术有限公司）；Imperial College London（帝国理工学院）
The ability to generate synthetic medical images is useful for data augmentation, domain transfer, and out-of-distribution detection. However, generating realistic, high-resolution medical images is challenging, particularly for Full Field Digital Mammograms (FFDM), due to the textural heterogeneity, fine structural details and specific tissue properties. In this paper, we explore the use of progressively trained generative adversarial networks (GANs) to synthesize mammograms, overcoming the underlying instabilities when training such adversarial models. This work is the first to show that gen…
We use Topological Data Analysis tools for studying the inner organization of cells in segmented images of epithelial tissues. More specifically, for each segmented image, we compute different persistence barcodes, which codify lifetime of homology classes (persistent homology) along different filtrations (increasing nested sequences of simplicial complexes) that are built from the regions representing the cells in the tissue. We use a complete and well-grounded set of numerical variables over those persistence barcodes, also known as topological summaries. A novel combination of normalization…
3D-Guided Scalable Flow Matching for Generating Volumetric Tissue Spatial Transcriptomics from Serial Histology
Cambridge Stem Cell Institute, University of Cambridge（剑桥干细胞研究所，剑桥大学）；Cambridge Center for AI in Medicine, University of Cambridge（剑桥人工智能医学中心，剑桥大学）；Department of Medicine, University of Cambridge（剑桥大学医学系）；Computer Science Department, University of Oxford（牛津大学计算机科学系）；Computer Science Department, University of Helsinki（赫尔辛基大学计算机科学系）；Institute for Molecular Medicine Finland, University of Helsinki（芬兰分子医学研究所，赫尔辛基大学）；Wellcome Sanger Institute（沃森桑格研究所）
fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting
Department of Biomedical Engineering, University of Basel, Switzerland Department of Quantitative Biomedicine, University of Zurich, Switzerland Helmholtz AI, Helmholtz Zentrum München, Germany Department of Diagnostic；Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany TranslaTUM - Central Institute for Translational Cancer Research, Technical University of Munich, Germany
Linköping University（林雪平大学）；Center for Medical Image Science and Visualization (CMIV)（医学影像科学与可视化中心（CMIV））；Lund University（隆德大学）
Diffusion magnetic resonance imaging (diffusion MRI) is a non-invasive microstructure assessment technique. Scalar measures, such as FA (fractional anisotropy) and MD (mean diffusivity), quantifying micro-structural tissue properties can be obtained using diffusion models and data processing pipelines. However, it is costly and time consuming to collect high quality diffusion data. Here, we therefore demonstrate how Generative Adversarial Networks (GANs) can be used to generate synthetic diffusion scalar measures from structural T1-weighted images in a single optimized step. Specifically, we t…
Digitization of histopathology slides has led to several advances, from easy data sharing and collaborations to the development of digital diagnostic tools. Deep learning (DL) methods for classification and detection have shown great potential, but often require large amounts of training data that are hard to collect, and annotate. For many cancer types, the scarceness of data creates barriers for training DL models. One such scenario relates to detecting tumor metastasis in lymph node tissue, where the low ratio of tumor to non-tumor cells makes the diagnostic task hard and time-consuming. DL…
Multiple sclerosis (MS) is a demyelinating disease of the central nervous system (CNS). A reliable measure of the tissue myelin content is therefore essential for the understanding of the physiopathology of MS, tracking progression and assessing treatment efficacy. Positron emission tomography (PET) with $[^{11} \mbox{C}] \mbox{PIB}$ has been proposed as a promising biomarker for measuring myelin content changes in-vivo in MS. However, PET imaging is expensive and invasive due to the injection of a radioactive tracer. On the contrary, magnetic resonance imaging (MRI) is a non-invasive, widely …
Eindhoven University of Technology（埃因霍温理工大学）；Philips Research Laboratories（飞利浦研究实验室）；Philips Healthcare（飞利浦医疗）
Generative adversarial networks (GANs) have provided promising data enrichment solutions by synthesizing high-fidelity images. However, generating large sets of labeled images with new anatomical variations remains unexplored. We propose a novel method for synthesizing cardiac magnetic resonance (CMR) images on a population of virtual subjects with a large anatomical variation, introduced using the 4D eXtended Cardiac and Torso (XCAT) computerized human phantom. We investigate two conditional image synthesis approaches grounded on a semantically-consistent mask-guided image generation techniqu…
ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）；IBM Research（IBM研究院）；IRCCS-Fondazione Pascale（IRCCS-帕斯卡尔基金会）；ICAR-CNR（意大利国家研究理事会ICAR研究所）
Cancer diagnosis, prognosis, and therapeutic response prediction are heavily influenced by the relationship between the histopathological structures and the function of the tissue. Recent approaches acknowledging the structure-function relationship, have linked the structural and spatial patterns of cell organization in tissue via cell-graphs to tumor grades. Though cell organization is imperative, it is insufficient to entirely represent the histopathological structure. We propose a novel hierarchical cell-to-tissue-graph (HACT) representation to improve the structural depiction of the tissue…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Magnetic Resonance (MR) Imaging and Computed Tomography (CT) are the primary diagnostic imaging modalities quite frequently used for surgical planning and analysis. A general problem with medical imaging is that the acquisition process is quite expensive and time-consuming. Deep learning techniques like generative adversarial networks (GANs) can help us to leverage the possibility of an image to image translation between multiple imaging modalities, which in turn helps in saving time and cost. These techniques will help to conduct surgical planning under CT with the feedback of MRI information…
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Heidelberg University（海德堡大学）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Cancer Research UK Cambridge Institute, University of Cambridge（剑桥大学英国癌症研究中心剑桥研究所）
Photoacoustic tomography (PAT) has the potential to recover morphological and functional tissue properties with high spatial resolution. However, previous attempts to solve the optical inverse problem with supervised machine learning were hampered by the absence of labeled reference data. While this bottleneck has been tackled by simulating training data, the domain gap between real and simulated images remains an unsolved challenge. We propose a novel approach to PAT image synthesis that involves subdividing the challenge of generating plausible simulations into two disjoint problems: (1) Pro…
Roche Innovation Center Munich（慕尼黑罗氏创新中心）；Technische Universität München（慕尼黑工业大学）
Style transfer is a field with growing interest and use cases in deep learning. Recent work has shown Generative Adversarial Networks(GANs) can be used to create realistic images of virtually stained slide images in digital pathology with clinically validated interpretability. Digital pathology images are typically of extremely high resolution, making tilewise analysis necessary for deep learning applications. It has been shown that image generators with instance normalization can cause a tiling artifact when a large image is reconstructed from the tilewise analysis. We introduce a novel perce…
Nanjing University of Information Science and Technology（南京信息工程大学）；Radboud University Medical Center（拉德堡德大学医学中心）；Shandong University (Weihai)（山东大学（威海））
Missing scans are inevitable in longitudinal studies due to either subject dropouts or failed scans. In this paper, we propose a deep learning framework to predict missing scans from acquired scans, catering to longitudinal infant studies. Prediction of infant brain MRI is challenging owing to the rapid contrast and structural changes particularly during the first year of life. We introduce a trustworthy metamorphic generative adversarial network (MGAN) for translating infant brain MRI from one time-point to another. MGAN has three key features: (i) Image translation leveraging spatial and fre…
The introduction of surgical robots brought about advancements in surgical procedures. The applications of remote telesurgery range from building medical clinics in underprivileged areas, to placing robots abroad in military hot-spots where accessibility and diversity of medical experience may be limited. Poor wireless connectivity may result in a prolonged delay, referred to as latency, between a surgeon's input and action a robot takes. In surgery, any micro-delay can injure a patient severely and in some cases, result in fatality. One was to increase safety is to mitigate the effects of lat…
IBM Research（IBM研究院）；ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）；IRCCS-Fondazione Pascale（帕斯卡尔IRCCS基金会）；ICAR-CNR（意大利国家研究委员会ICAR研究所）；Aurigen（Aurigen公司）；CHUV（洛桑大学医院）
Cancer diagnosis, prognosis, and therapy response predictions from tissue specimens highly depend on the phenotype and topological distribution of constituting histological entities. Thus, adequate tissue representations for encoding histological entities is imperative for computer aided cancer patient care. To this end, several approaches have leveraged cell-graphs that encode cell morphology and organization to denote the tissue information. These allow for utilizing machine learning to map tissue representations to tissue functionality to help quantify their relationship. Though cellular in…
IBM Research（IBM研究院）；EPFL（洛桑联邦理工学院）；ETH（苏黎世联邦理工学院）
Advances in entity-graph based analysis of histopathology images have brought in a new paradigm to describe tissue composition, and learn the tissue structure-to-function relationship. Entity-graphs offer flexible and scalable representations to characterize tissue organization, while allowing the incorporation of prior pathological knowledge to further support model interpretability and explainability. However, entity-graph analysis requires prerequisites for image-to-graph translation and knowledge of state-of-the-art machine learning algorithms applied to graph-structured data, which can po…
SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome
Wellcome Sanger Institute（沃森桑格研究所）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；Institute of AI for Health（人工智能与健康研究所）；Cambridge Stem Cell Institute（剑桥干细胞研究所）
Inception Institute of Artificial Intelligence（起源人工智能研究院）
Gleason grading from histopathology images is essential for accurate prostate cancer (PCa) diagnosis. Since such images are obtained after invasive tissue resection quick diagnosis is challenging under the existing paradigm. We propose a method to predict Gleason grades from magnetic resonance (MR) images which are non-interventional and easily acquired. We solve the problem in a generalized zero-shot learning (GZSL) setting since we may not access training images of every disease grade. Synthetic MRI feature vectors of unseen grades (classes) are generated by exploiting Gleason grades' ordere…
Nanyang Technological University（南洋理工大学）；Institute of Bioengineering and Bioimaging, A*STAR（A*STAR生物工程与生物成像研究所）
Staining is critical to cell imaging and medical diagnosis, which is expensive, time-consuming, labor-intensive, and causes irreversible changes to cell tissues. Recent advances in deep learning enabled digital staining via supervised model training. However, it is difficult to obtain large-scale stained/unstained cell image pairs in practice, which need to be perfectly aligned with the supervision. In this work, we propose a novel unsupervised deep learning framework for the digital staining of cell images using knowledge distillation and generative adversarial networks (GANs). A teacher mode…
Laboratory of Image Synthesis and Analysis (LISA)（图像合成与分析实验室（LISA））；Université Libre de Bruxelles (ULB)（布鲁塞尔自由大学（ULB））；DIAPath, Center for Microscopy and Molecular Imaging (CMMI)（DIAPath显微镜与分子成像中心（CMMI））；Environmental HydroAcoustics Lab (EHL)（环境水声学实验室（EHL））
Characterizing the tissue morphology and anatomy of seagrasses is essential to predicting their acoustic behavior. In this pilot study, we use histology techniques and whole slide imaging (WSI) to describe the composition and topology of the aerenchyma of an entire leaf blade in an automatic way combining the advantages of X-ray microtomography and optical microscopy. Paraffin blocks are prepared in such a way that microtome slices contain an arbitrarily large number of cross sections distributed along the full length of a blade. The sample organization in the paraffin block coupled with whole…
According to the World Health Organization, breast cancer is the most common form of cancer in women. It is the second leading cause of death among women round the world, becoming the most fatal form of cancer. Mammographic image segmentation is a fundamental task to support image analysis and diagnosis, taking into account shape analysis of mammary lesions and their borders. However, mammogram segmentation is a very hard process, once it is highly dependent on the types of mammary tissues. In this work we present a new semi-supervised segmentation algorithm based on the modification of the Gr…
Brigham and Women's Hospital（布莱根和妇女医院）；Harvard Medical School（哈佛医学院）；Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）
Histology-based grade classification is clinically important for many cancer types in stratifying patients distinct treatment groups. In prostate cancer, the Gleason score is a grading system used to measure the aggressiveness of prostate cancer from the spatial organization of cells and the distribution of glands. However, the subjective interpretation of Gleason score often suffers from large interobserver and intraobserver variability. Previous work in deep learning-based objective Gleason grading requires manual pixel-level annotation. In this work, we propose a weakly-supervised approach …
