Multi-focus image fusion technologies compress different focus depth images into an image in which most objects are in focus. However, although existing image fusion techniques, including traditional algorithms and deep learning-based algorithms, can generate high-quality fused images, they need multiple images with different focus depths in the same field of view. This criterion may not be met in some cases where time efficiency is required or the hardware is insufficient. The problem is especially prominent in large-size whole slide images. This paper focused on the multi-focus image fusion of cytopathological digital slide images, and proposed a novel method for generating fused images from single-focus or few-focus images based on conditional generative adversarial network (GAN). Through the adversarial learning of the generator and discriminator, the method is capable of generating fused images with clear textures and large depth of field. Combined with the characteristics of cytopathological images, this paper designs a new generator architecture combining U-Net and DenseBlock, which can effectively improve the network's receptive field and comprehensively encode image features. Meanwhile, this paper develops a semantic segmentation network that identifies the blurred regions in cytopathological images. By integrating the network into the generative model, the quality of the generated fused images is effectively improved. Our method can generate fused images from only single-focus or few-focus images, thereby avoiding the problem of collecting multiple images of different focus depths with increased time and hardware costs. Furthermore, our model is designed to learn the direct mapping of input source images to fused images without the need to manually design complex activity level measurements and fusion rules as in traditional methods.
Beijing University of Post and Telecommunication（北京邮电大学）；China-Japanese Friendship Hospital（中日友好医院）；Haohandata Technology Co.（好汉数据科技公司）
In this paper, we propose a novel method for highly efficient follicular segmentation of thyroid cytopathological WSIs. Firstly, we propose a hybrid segmentation architecture, which integrates a classifier into Deeplab V3 by adding a branch. A large amount of the WSI segmentation time is saved by skipping the irrelevant areas using the classification branch. Secondly, we merge the low scale fine features into the original atrous spatial pyramid pooling (ASPP) in Deeplab V3 to accurately represent the details in cytopathological images. Thirdly, our hybrid model is trained by a criterion-orient…
University of California, Davis（加州大学戴维斯分校）；University of San Francisco Data Institute（旧金山大学数据研究所）；Wicklow AI in Medicine Research Initiative（威克洛医学人工智能研究计划）；University of Texas, San Antonio Health（德克萨斯大学圣安东尼奥健康分校）
MUSE is a novel slide-free imaging technique for histological examination of tissues that can serve as an alternative to traditional histology. In order to bridge the gap between MUSE and traditional histology, we aim to convert MUSE images to resemble authentic hematoxylin- and eosin-stained (H&E) images. We evaluated four models: a non-machine-learning-based color-mapping unmixing-based tool, CycleGAN, DualGAN, and GANILLA. CycleGAN and GANILLA provided visually compelling results that appropriately transferred H&E style and preserved MUSE content. Based on training an automated critic on re…
University of Bayreuth（拜罗伊特大学）；Bundeswehr University Munich（慕尼黑联邦国防军大学）；LMU Munich（慕尼黑大学）
We investigate how multiple sliders with and without feedforward visualizations influence users' control of generative models. In an online study (N=138), we collected a dataset of people interacting with a generative adversarial network (StyleGAN2) in an image reconstruction task. We found that more control dimensions (sliders) significantly increase task difficulty and user actions. Visual feedforward partly mitigates this by enabling more goal-directed interaction. However, we found no evidence of faster or more accurate task performance. This indicates a tradeoff between feedforward detail…
Queen Mary University of London（伦敦玛丽女王大学）；Karolinska University Hospital（卡罗林斯卡大学医院）
Histopathological cancer diagnosis is based on visual examination of stained tissue slides. Hematoxylin and eosin (H\&E) is a standard stain routinely employed worldwide. It is easy to acquire and cost effective, but cells and tissue components show low-contrast with varying tones of dark blue and pink, which makes difficult visual assessments, digital image analysis, and quantifications. These limitations can be overcome by IHC staining of target proteins of the tissue slide. IHC provides a selective, high-contrast imaging of cells and tissue components, but their use is largely limited by a …
Roche Innovation Center Munich（慕尼黑罗氏创新中心）；Technische Universität München（慕尼黑工业大学）
Style transfer is a field with growing interest and use cases in deep learning. Recent work has shown Generative Adversarial Networks(GANs) can be used to create realistic images of virtually stained slide images in digital pathology with clinically validated interpretability. Digital pathology images are typically of extremely high resolution, making tilewise analysis necessary for deep learning applications. It has been shown that image generators with instance normalization can cause a tiling artifact when a large image is reconstructed from the tilewise analysis. We introduce a novel perce…
Beyond Sliders: Mastering the Art of Diffusion-based Image Manipulation
FUYAO University Of Science And Technology（福尧大学科学技术学院）；University of Science and Technology of China（中国科学技术大学）；Anhui Province Key Laboratory of Digital Security（安徽省数字安全重点实验室）；Alibaba TongYi Lab（阿里巴巴通义实验室）
Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland（芬兰奥卢大学机器视觉与信号分析中心，芬兰奥卢）
Image-to-image (i2i) translation is the dense regression problem of learning how to transform an input image into an output using aligned image pairs. Remarkable progress has been made in i2i translation with the advent of Deep Convolutional Neural Networks (DCNNs) and particular using the learning paradigm of Generative Adversarial Networks (GANs). In the absence of paired images, i2i translation is tackled with one or multiple domain transformations (i.e., CycleGAN, StarGAN etc.). In this paper, we study a new problem, that of image-to-image translation, under a set of continuous parameters …
Salzburg University of Applied Sciences（萨尔茨堡应用科学大学）
Image analysis in the field of digital pathology has recently gained increased popularity. The use of high-quality whole slide scanners enables the fast acquisition of large amounts of image data, showing extensive context and microscopic detail at the same time. Simultaneously, novel machine learning algorithms have boosted the performance of image analysis approaches. In this paper, we focus on a particularly powerful class of architectures, called Generative Adversarial Networks (GANs), applied to histological image data. Besides improving performance, GANs also enable application scenarios…
The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院）
Intratumoral nanoparticles (NPs) distribution is critical for the success of nanomedicine in imaging and treatment, but computational models to describe the NPs distribution remain unavailable due to the complex tumor-nano interactions. Here, we develop a Generative Adversarial Network for Distribution Analysis (GANDA) to describe and conditionally generates the intratumoral quantum dots (QDs) distribution after i.v. injection. This deep generative model is trained automatically by 27 775 patches of tumor vessels and cell nuclei decomposed from whole-slide images of 4T1 breast cancer sections.…
University College London（伦敦大学学院）；Brigham and Women's Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Broad Institute of Harvard and MIT（哈佛-麻省理工博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；University of Warwick（华威大学）；University Hospitals Coventry and Warwickshire NHS Trust（考文垂与沃里克郡国民保健服务信托大学医院）；The Alan Turing Institute（艾伦·图灵研究所）
Multiple Instance Learning (MIL) is a widely employed framework for learning on gigapixel whole-slide images (WSIs) from WSI-level annotations. In most MIL based analytical pipelines for WSI-level analysis, the WSIs are often divided into patches and deep features for patches (i.e., patch embeddings) are extracted prior to training to reduce the overall computational cost and cope with the GPUs' limited RAM. To overcome this limitation, we present EmbAugmenter, a data augmentation generative adversarial network (DA-GAN) that can synthesize data augmentations in the embedding space rather than …
University of Toronto（多伦多大学）；University of Alberta（阿尔伯塔大学）
Computational histopathology image diagnosis becomes increasingly popular and important, where images are segmented or classified for disease diagnosis by computers. While pathologists do not struggle with color variations in slides, computational solutions usually suffer from this critical issue. To address the issue of color variations in histopathology images, this study proposes two stain style transfer models, SSIM-GAN and DSCSI-GAN, based on the generative adversarial networks. By cooperating structural preservation metrics and feedback of an auxiliary diagnosis net in learning, medical-…
Technical University Munich（慕尼黑工业大学）；Helmholtz AI（亥姆霍兹人工智能研究所）；Munich School of Data Science (MuDS)（慕尼黑数据科学学院）；ContextVision AB（ContextVision公司）；Institute for Computational Biology, HelmholtzZentrum Munich（慕尼黑亥姆霍兹中心计算生物学研究所）
In digital pathology, different staining procedures and scanners cause substantial color variations in whole-slide images (WSIs), especially across different laboratories. These color shifts result in a poor generalization of deep learning-based methods from the training domain to external pathology data. To increase test performance, stain normalization techniques are used to reduce the variance between training and test domain. Alternatively, color augmentation can be applied during training leading to a more robust model without the extra step of color normalization at test time. We propose…
F2FLDM: Latent Diffusion Models with Histopathology Pre-Trained Embeddings for Unpaired Frozen Section to FFPE Translation
University of Utah（犹他大学）；The Catholic University of Korea College of Medicine（韩国天主教大学医学院）；Huntsman Cancer Institute（亨茨曼癌症研究所）；Scientific Computing and Imaging Institute（科学计算与成像研究所）
Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
We develop a new method for portrait image editing, which supports fine-grained editing of geometries, colors, lights and shadows using a single neural network model. We adopt a novel asymmetric conditional GAN architecture: the generators take the transformed conditional inputs, such as edge maps, color palette, sliders and masks, that can be directly edited by the user; the discriminators take the conditional inputs in the way that can guide controllable image generation more effectively. Taking color editing as an example, we feed color palettes (which can be edited easily) into the generat…
VinAI Research（VinAI研究院）；FPT University（FPT大学）；University of Arkansas（阿肯色大学）
Image synthesis from corrupted contrasts increases the diversity of diagnostic information available for many neurological diseases. Recently the image-to-image translation has experienced significant levels of interest within medical research, beginning with the successful use of the Generative Adversarial Network (GAN) to the introduction of cyclic constraint extended to multiple domains. However, in current approaches, there is no guarantee that the mapping between the two image domains would be unique or one-to-one. In this paper, we introduce a novel approach to unpaired image-to-image tr…
The University of Tokyo（东京大学）；National Taiwan University（台湾大学）
Generative image modeling techniques such as GAN demonstrate highly convincing image generation result. However, user interaction is often necessary to obtain the desired results. Existing attempts add interactivity but require either tailored architectures or extra data. We present a human-in-the-optimization method that allows users to directly explore and search the latent vector space of generative image modeling. Our system provides multiple candidates by sampling the latent vector space, and the user selects the best blending weights within the subspace using multiple sliders. In additio…
Building Trust in Virtual Immunohistochemistry: Automated Assessment of Image Quality
Kahlert School of Computing, University of Utah（犹他大学卡勒特计算学院）；Department of Internal Medicine, Division of Epidemiology, University of Utah（犹他大学内科学系流行病学部门）；Department of Pathology, University of Utah（犹他大学病理学系）；ARUP Laboratories（ARUP实验室）
