Convolutional Neural Networks (CNN) based image reconstruction methods have been intensely used for X-ray computed tomography (CT) reconstruction applications. Despite great success, good performance of this data-based approach critically relies on a representative big training data set and a dense convoluted deep network. The indiscriminating convolution connections over all dense layers could be prone to over-fitting, where sampling biases are wrongly integrated as features for the reconstruction. In this paper, we report a robust hierarchical synthesis reconstruction approach, where training data is pre-processed to separate the information on the domains where sampling biases are suspected. These split bands are then trained separately and combined successively through a hierarchical synthesis network. We apply the hierarchical synthesis reconstruction for two important and classical tomography reconstruction scenarios: the spares-view reconstruction and the phase reconstruction. Our simulated and experimental results show that comparable or improved performances are achieved with a dramatic reduction of network complexity and computational cost. This method can be generalized to a wide range of applications including material characterization, in-vivo monitoring and dynamic 4D imaging.
University of Sydney（悉尼大学）；Shanghai Jiao Tong University（上海交通大学）；Royal Prince Alfred Hospital（皇家阿尔弗雷德王子医院）
Positron emission tomography (PET) image synthesis plays an important role, which can be used to boost the training data for computer aided diagnosis systems. However, existing image synthesis methods have problems in synthesizing the low resolution PET images. To address these limitations, we propose multi-channel generative adversarial networks (M-GAN) based PET image synthesis method. Different to the existing methods which rely on using low-level features, the proposed M-GAN is capable to represent the features in a high-level of semantic based on the adversarial learning concept. In addit…
Tianjin University（天津大学）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；La Trobe University（乐卓博大学）；Tianjin University of Traditional Chinese Medicine（天津中医药大学）
The insufficiency of annotated medical imaging scans for cancer makes it challenging to train and validate data-hungry deep learning models in precision oncology. We propose a new richer generative adversarial network for free-form 3D tumor/lesion synthesis in computed tomography (CT) images. The network is composed of a new richer convolutional feature enhanced dilated-gated generator (RicherDG) and a hybrid loss function. The RicherDG has dilated-gated convolution layers to enable tumor-painting and to enlarge perceptive fields; and it has a novel richer convolutional feature association bra…
Deep learning is a popular and powerful tool in computed tomography (CT) image processing such as organ segmentation, but its requirement of large training datasets remains a challenge. Even though there is a large anatomical variability for children during their growth, the training datasets for pediatric CT scans are especially hard to obtain due to risks of radiation to children. In this paper, we propose a method to conditionally synthesize realistic pediatric CT images using a new auxiliary classifier generative adversarial network (ACGAN) architecture by taking age information into accou…
Indian Institute of Information Technology, Sri City（印度信息技术学院，斯里城）
Image to Image Translation (I2I) is a challenging computer vision problem used in numerous domains for multiple tasks. Recently, ophthalmology became one of the major fields where the application of I2I is increasing rapidly. One such application is the generation of synthetic retinal optical coherence tomographic (OCT) scans. Existing I2I methods require training of multiple models to translate images from normal scans to a specific pathology: limiting the use of these models due to their complexity. To address this issue, we propose an unsupervised multi-domain I2I network with pre-trained s…
Imagia Inc.（Imagia公司）；Quebec AI Institute (Mila)（魁北克人工智能研究所（Mila））；McGill University（麦吉尔大学）；University of Waterloo（滑铁卢大学）
Diversity in data is critical for the successful training of deep learning models. Leveraged by a recurrent generative adversarial network, we propose the CT-SGAN model that generates large-scale 3D synthetic CT-scan volumes ($\geq 224\times224\times224$) when trained on a small dataset of chest CT-scans. CT-SGAN offers an attractive solution to two major challenges facing machine learning in medical imaging: a small number of given i.i.d. training data, and the restrictions around the sharing of patient data preventing to rapidly obtain larger and more diverse datasets. We evaluate the fideli…
This paper presents a novel method to deal with the challenging task of generating photographic images conditioned on semantic image descriptions. Our method introduces accompanying hierarchical-nested adversarial objectives inside the network hierarchies, which regularize mid-level representations and assist generator training to capture the complex image statistics. We present an extensile single-stream generator architecture to better adapt the jointed discriminators and push generated images up to high resolutions. We adopt a multi-purpose adversarial loss to encourage more effective image…
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）；Osaka University Graduate School of Medicine（大阪大学大学院医学系研究科）；Ehime University Graduate School of Medicine（爱媛大学大学院医学系研究科）
We propose a method for estimating the bone mineral density (BMD) from a plain x-ray image. Dual-energy X-ray absorptiometry (DXA) and quantitative computed tomography (QCT) provide high accuracy in diagnosing osteoporosis; however, these modalities require special equipment and scan protocols. Measuring BMD from an x-ray image provides an opportunistic screening, which is potentially useful for early diagnosis. The previous methods that directly learn the relationship between x-ray images and BMD require a large training dataset to achieve high accuracy because of large intensity variations i…
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）；Huashan Hospital, Fudan University（复旦大学附属华山医院）；Fudan University（复旦大学）；Jing’an District Centre Hospital of Shanghai（上海市静安区中心医院）
Ischemic stroke lesion segmentation from Computed Tomography Perfusion (CTP) images is important for accurate diagnosis of stroke in acute care units. However, it is challenged by low image contrast and resolution of the perfusion parameter maps, in addition to the complex appearance of the lesion. To deal with this problem, we propose a novel framework based on synthesized pseudo Diffusion-Weighted Imaging (DWI) from perfusion parameter maps to obtain better image quality for more accurate segmentation. Our framework consists of three components based on Convolutional Neural Networks (CNNs) a…
We propose a novel hierarchical approach for text-to-image synthesis by inferring semantic layout. Instead of learning a direct mapping from text to image, our algorithm decomposes the generation process into multiple steps, in which it first constructs a semantic layout from the text by the layout generator and converts the layout to an image by the image generator. The proposed layout generator progressively constructs a semantic layout in a coarse-to-fine manner by generating object bounding boxes and refining each box by estimating object shapes inside the box. The image generator synthesi…
Shanghai Jiao Tong University（上海交通大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Generating diverse and natural human motion is one of the long-standing goals for creating intelligent characters in the animated world. In this paper, we propose a self-supervised method for generating long-range, diverse and plausible behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by decomposing a long-range generation task in a hierarchical manner. Given the starting and ending states, a memory bank is used to retrieve motion references as source material for short-range clip generation. We first propose to explicitly disentangle the p…
MEDPSeg: Hierarchical polymorphic multitask learning for the segmentation of ground-glass opacities, consolidation, and pulmonary structures on computed tomography
School of Electrical and Computer Engineering, Universidade Estadual de Campinas（坎皮纳斯州立大学电气与计算机工程学院）；University of Iowa（爱荷华大学）
Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers
School of Systems and Computing University of New South Wales（系统与计算学院 新南威尔士大学）；School of Engineering Australian National University（工程学院 澳大利亚国立大学）；School of Business University of New South Wales（商学院 新南威尔士大学）；Australian Institute for Machine Learning University of Adelaide（机器学习研究所 阿德莱德大学）；School of Computer Science and Engineering University of New South Wales（计算机科学与工程学院 新南威尔士大学）；School of Computer Science University of Technology Sydney（计算机科学学院 技术大学悉尼）
HUG-VAS: A Hierarchical NURBS-Based Generative Model for Aortic Geometry Synthesis and Controllable Editing
Department of Aerospace and Mechanical Engineering, University of Notre Dame（notre dame 大学航空航天与机械工程系）；Sibley School of Mechanical and Aerospace Engineering, Cornell University（cornell 大学机械与航空航天工程学院）；Department of Applied and Computational Mathematics and Statistics, University of Notre Dame（notre dame 大学应用与计算数学与统计学系）
FreeTumor: Large-Scale Generative Tumor Synthesis in Computed Tomography Images for Improving Tumor Recognition
The Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯AI实验室）；Harvard University（哈佛大学）；The Chinese University of Hong Kong（香港中文大学）；The Third Affiliated Hospital of Southern Medical University（南方医科大学第三附属医院）；Sun Yat-Sen Memorial Hospital, Sun Yat-Sen University（中山大学孙逸仙纪念医院）；The University of Hong Kong（香港大学）；Shenzhen-Hong Kong Collaborative Innovation Research Institute（深港产学研协同创新研究院）
Nanchang Hangkong University（南昌航空大学）；Dalian University of Technology（大连理工大学）；Cardiff University（卡迪夫大学）
Most of the existing audio-driven 3D facial animation methods suffered from the lack of detailed facial expression and head pose, resulting in unsatisfactory experience of human-robot interaction. In this paper, a novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. To synthesize real and detailed expression, a hierarchical decomposition strategy is proposed to encode the audio signal into both a global latent feature and a local vertex-wise control feature. Then the local and global audio features combined with vertex spati…
University of Campinas（坎皮纳斯大学）；Innopolis University（因诺波利斯大学）
Anomaly detection suffers from unbalanced data since anomalies are quite rare. Synthetically generated anomalies are a solution to such ill or not fully defined data. However, synthesis requires an expressive representation to guarantee the quality of the generated data. In this paper, we propose a two-level hierarchical latent space representation that distills inliers' feature-descriptors (through autoencoders) into more robust representations based on a variational family of distributions (through a variational autoencoder) for zero-shot anomaly generation. From the learned latent distribut…
This fourth and last tome is focusing on describing the envisioned works for a project that has been presented in the preceding tome. It is about a new approach dedicated to the coding of still and moving pictures, trying to bridge the MPEG-4 and MPEG-7 standard bodies. The aim of this project is to define the principles of self-descriptive video coding. In order to establish them, the document is composed in five chapters that describe the various envisioned techniques for developing such a new approach in visual coding: - image segmentation, - computation of visual descriptors, - computation…
KAUST（阿卜杜拉国王科技大学）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）
In this paper, we present Neural Adaptive Tomography (NeAT), the first adaptive, hierarchical neural rendering pipeline for multi-view inverse rendering. Through a combination of neural features with an adaptive explicit representation, we achieve reconstruction times far superior to existing neural inverse rendering methods. The adaptive explicit representation improves efficiency by facilitating empty space culling and concentrating samples in complex regions, while the neural features act as a neural regularizer for the 3D reconstruction. The NeAT framework is designed specifically for the …
School of Computing Science（计算科学学院）；Google（谷歌）；Simon Fraser University（西蒙弗雷泽大学）
A persistent challenge in conditional image synthesis has been to generate diverse output images from the same input image despite only one output image being observed per input image. GAN-based methods are prone to mode collapse, which leads to low diversity. To get around this, we leverage Implicit Maximum Likelihood Estimation (IMLE) which can overcome mode collapse fundamentally. IMLE uses the same generator as GANs but trains it with a different, non-adversarial objective which ensures each observed image has a generated sample nearby. Unfortunately, to generate high-fidelity images, prio…
Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
Harvard Medical School and the Brigham and Women’s Hospital（哈佛医学院和布里洛妇产科医院）；MIND Team, Inria Saclay, Université Paris-Saclay, Palaiseau, France and Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière（MIND团队、Inria Saclay、巴黎萨克雷大学、Palaiseau，法国以及索邦大学、巴黎脑研究所-ICM、CNRS、Inria、Inserm、AP-HP、皮蒂埃萨勒普蒂耶医院）；Massachusetts Institute of Technology（麻省理工学院）
From Baseline to Follow-Up: Counterfactual Spine DXA Image Synthesis in UK Biobank Using a Causal Hierarchical Variational Autoencoder
School of Electronics and Computer Science（电子与计算机科学学院）；University of Southampton（萨塞克斯大学）；MRC Lifecourse Epidemiology Centre（英国医学研究理事会生命周期流行病学中心）；University of Southampton, Southampton General Hospital（萨塞克斯大学索马塞特医院）；Computer Science University of Southampton（计算机科学萨塞克斯大学）
A Label-Free High-Precision Residual Moveout Picking Method for Travel Time Tomography based on Deep Learning
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；China National Petroleum Corp Bureau of Geophysical Prospecting Inc, Geophysical Technology Research Center（中国石油集团东方地球物理勘探有限责任公司物探技术研究中心）
Samsung Medical Center（三星医疗中心）；Sungkyunkwan University School of Medicine（成均馆大学医学院）；Sungkyunkwan University（成均馆大学）
Compared with chest X-ray (CXR) imaging, which is a single image projected from the front of the patient, chest digital tomosynthesis (CDTS) imaging can be more advantageous for lung lesion detection because it acquires multiple images projected from multiple angles of the patient. Various clinical comparative analysis and verification studies have been reported to demonstrate this, but there were no artificial intelligence (AI)-based comparative analysis studies. Existing AI-based computer-aided detection (CAD) systems for lung lesion diagnosis have been developed mainly based on CXR images; …
H2ASeg: Hierarchical Adaptive Interaction and Weighting Network for Tumor Segmentation in PET/CT Images
School of Science, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）理学院）；School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）
King’s College London（伦敦国王学院）；University College London（伦敦大学学院）
The ability to synthesise Computed Tomography images - commonly known as pseudo CT, or pCT - from MRI input data is commonly assessed using an intensity-wise similarity, such as an L2-norm between the ground truth CT and the pCT. However, given that the ultimate purpose is often to use the pCT as an attenuation map ($μ$-map) in Positron Emission Tomography Magnetic Resonance Imaging (PET/MRI), minimising the error between pCT and CT is not necessarily optimal. The main objective should be to predict a pCT that, when used as $μ$-map, reconstructs a pseudo PET (pPET) which is as close as possibl…
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Rochester（罗切斯特大学）；University of Science and Technology of China（中国科学技术大学）
It is a long-standing challenge to reconstruct Cone Beam Computed Tomography (CBCT) of the lung under respiratory motion. This work takes a step further to address a challenging setting in reconstructing a multi-phase}4D lung image from just a single}3D CBCT acquisition. To this end, we introduce REpiratory-GAted Synthesis of views, or REGAS. REGAS proposes a self-supervised method to synthesize the undersampled tomographic views and mitigate aliasing artifacts in reconstructed images. This method allows a much better estimation of between-phase Deformation Vector Fields (DVFs), which are used…
Harbin Engineering University（哈尔滨工程大学）；NC State University（北卡罗来纳州立大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
This paper proposes to learn hierarchical compositional AND-OR model for interpretable image synthesis by sparsifying the generator network. The proposed method adopts the scene-objects-parts-subparts-primitives hierarchy in image representation. A scene has different types (i.e., OR) each of which consists of a number of objects (i.e., AND). This can be recursively formulated across the scene-objects-parts-subparts hierarchy and is terminated at the primitive level (e.g., wavelets-like basis). To realize this AND-OR hierarchy in image synthesis, we learn a generator network that consists of t…
ARC Training Centre for Innovative BioEngineering（ARC创新生物工程培训中心）
Distant metastases (DM) refer to the dissemination of tumors, usually, beyond the organ where the tumor originated. They are the leading cause of death in patients with soft-tissue sarcomas (STSs). Positron emission tomography-computed tomography (PET-CT) is regarded as the imaging modality of choice for the management of STSs. It is difficult to determine from imaging studies which STS patients will develop metastases. 'Radiomics' refers to the extraction and analysis of quantitative features from medical images and it has been employed to help identify such tumors. The state-of-the-art in ra…
HiFi-Syn: Hierarchical Granularity Discrimination for High-Fidelity Synthesis of MR Images with Structure Preservation
Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院）；College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院健康科学技术学院）；Clinical Neuroscience Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属瑞金医院临床神经科学中心）；National Engineering Research Center of Advanced Magnetic Resonance Technologies for Diagnosis and Therapy, Shanghai Jiao Tong University（上海交通大学先进磁共振诊断与治疗技术国家工程研究中心）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
University of Amsterdam（阿姆斯特丹大学）；Shanghai Jiao Tong University（上海交通大学）
This paper strives to generate a synthetic computed tomography (CT) image from a magnetic resonance (MR) image. The synthetic CT image is valuable for radiotherapy planning when only an MR image is available. Recent approaches have made large strides in solving this challenging synthesis problem with convolutional neural networks that learn a mapping from MR inputs to CT outputs. In this paper, we find that all existing approaches share a common limitation: reconstruction breaks down in and around the high-frequency parts of CT images. To address this common limitation, we introduce frequency-…
MedLoRD: A Medical Low-Resource Diffusion Model for High-Resolution 3D CT Image Synthesis
Institute for Artificial Intelligence in Cardiovascular Medicine（心血管医学人工智能研究所）；Heidelberg University Hospital（海德堡大学医院）；AI Health Innovation Cluster (AIH)（人工智能健康创新集群）；Heidelberg Faculty of Medicine（海德堡医学院）；German Centre for Cardiovascular Research (DZHK)（德国心脏病研究中心）；University Medical Center Mannheim（曼海姆大学医学中心）；Clinic for Diagnostic and Interventional Radiology（诊断与介入放射科诊所）
NVIDIA（英伟达）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with…
