DeepFakes are synthetic videos generated by swapping a face of an original image with the face of somebody else. In this paper, we describe our work to develop general, deep learning-based models to classify DeepFake content. We propose a novel framework for using Generative Adversarial Network (GAN)-based models, we call MRI-GAN, that utilizes perceptual differences in images to detect synthesized videos. We test our MRI-GAN approach and a plain-frames-based model using the DeepFake Detection Challenge Dataset. Our plain frames-based-model achieves 91% test accuracy and a model which uses our MRI-GAN framework with Structural Similarity Index Measurement (SSIM) for the perceptual differences achieves 74% test accuracy. The results of MRI-GAN are preliminary and may be improved further by modifying the choice of loss function, tuning hyper-parameters, or by using a more advanced perceptual similarity metric.
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Whirlpool（惠而浦公司）
Capturing high-resolution magnetic resonance (MR) images is a time consuming process, which makes it unsuitable for medical emergencies and pediatric patients. Low-resolution MR imaging, by contrast, is faster than its high-resolution counterpart, but it compromises on fine details necessary for a more precise diagnosis. Super-resolution (SR), when applied to low-resolution MR images, can help increase their utility by synthetically generating high-resolution images with little additional time. In this paper, we present a SR technique for MR images that is based on generative adversarial netwo…
Disruptive Attacks on Face Swapping via Low-Frequency Perceptual Perturbations
Department of Mathematics and Artificial Intelligence（数学与人工智能系）；Shandong Artificial Intelligence Institute（山东省人工智能研究所）；Qilu University of Technology（齐鲁大学）
Defending Deepfake via Texture Feature Perturbation
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Qilu University of Technology (Shandong Academy of Sciences)（计算机网络与信息安全重点实验室，教育部，齐鲁工业大学（山东省科学院））；Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)（山东省人工智能研究院，齐鲁工业大学（山东省科学院））；School of Computing, National University of Singapore（新加坡国立大学计算机学院）
Reconstruct or Generate: Exploring the Spectrum of Generative Modeling for Cardiac MRI
School of Computation, Information and Technology（计算信息学院）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；School of Medicine, Klinikum Rechts der Isar（医学院，莱茵河右岸医院）；Department of Quantitative Biomedicine（定量生物医学系）；University of Zurich（苏黎世大学）；Department of Computing（计算系）；Imperial College London（伦敦帝国理工学院）
Conditional Denoising Diffusion Model-Based Robust MR Image Reconstruction from Highly Undersampled Data
Department of Mathematics and Systems Engineering（数学与系统工程系）；Department of Biomedical Engineering & Science（生物医学工程与科学系）；Florida Institute of Technology（佛罗里达理工学院）；A. A. Martinos Center for Biomedical Imaging（A.A. Martinos生物医学成像中心）；Massachusetts General Hospital/Harvard Medical School（麻省总医院/哈佛医学院）；Department of Radiology（放射科）；Embry-Riddle Aeronautical University（艾姆伯-里尔航空大学）
Mayo Clinic（梅奥诊所）；University of Minnesota（明尼苏达大学）；Google LLC（谷歌有限责任公司）
There is a growing demand for high-resolution (HR) medical images in both the clinical and research applications. Image quality is inevitably traded off with the acquisition time for better patient comfort, lower examination costs, dose, and fewer motion-induced artifacts. For many image-based tasks, increasing the apparent resolution in the perpendicular plane to produce multi-planar reformats or 3D images is commonly used. Single image super-resolution (SR) is a promising technique to provide HR images based on unsupervised learning to increase resolution of a 2D image, but there are few rep…
Recent attempts at Super-Resolution for medical images used deep learning techniques such as Generative Adversarial Networks (GANs) to achieve perceptually realistic single image Super-Resolution. Yet, they are constrained by their inability to generalise to different scale factors. This involves high storage and energy costs as every integer scale factor involves a separate neural network. A recent paper has proposed a novel meta-learning technique that uses a Weight Prediction Network to enable Super-Resolution on arbitrary scale factors using only a single neural network. In this paper, we …
Imperial College London（伦敦帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）；Yantai University（烟台大学）；University of Cambridge（剑桥大学）；Beijing Institute of Technology（北京理工大学）
Magnetic Resonance Imaging (MRI) is a vital component of medical imaging. When compared to other image modalities, it has advantages such as the absence of radiation, superior soft tissue contrast, and complementary multiple sequence information. However, one drawback of MRI is its comparatively slow scanning and reconstruction compared to other image modalities, limiting its usage in some clinical applications when imaging time is critical. Traditional compressive sensing based MRI (CS-MRI) reconstruction can speed up MRI acquisition, but suffers from a long iterative process and noise-induce…
LDPM: Towards undersampled MRI reconstruction with MR-VAE and Latent Diffusion Prior
Shenzhen Technology University（深圳科技大学）；Shenzhen University（深圳大学）；Huawei（华为）；Nanjing University of Science and Technology（南京理工大学）；Qilu University of Technology（齐鲁工业大学）
This work is an update of a previous paper on the same topic published a few years ago. With the dramatic progress in generative modeling, a suite of new quantitative and qualitative techniques to evaluate models has emerged. Although some measures such as Inception Score, Frechet Inception Distance, Precision-Recall, and Perceptual Path Length are relatively more popular, GAN evaluation is not a settled issue and there is still room for improvement. Here, I describe new dimensions that are becoming important in assessing models (e.g. bias and fairness) and discuss the connection between GAN e…
University of Cambridge（剑桥大学）；Royal Brompton Hospital（皇家布朗普顿医院）；Imperial College London（帝国理工学院）
Single image super-resolution (SISR) is of great importance as a low-level computer vision task. The fast development of Generative Adversarial Network (GAN) based deep learning architectures realises an efficient and effective SISR to boost the spatial resolution of natural images captured by digital cameras. However, the SISR for medical images is still a very challenging problem. This is due to (1) compared to natural images, in general, medical images have lower signal to noise ratios, (2) GAN based models pre-trained on natural images may synthesise unrealistic patterns in medical images …
Imperial College London（伦敦帝国理工学院）；City University London（伦敦城市大学）；University of Lincoln（林肯大学）；University College London（伦敦大学学院）
Fast Magnetic Resonance Imaging (MRI) is highly in demand for many clinical applications in order to reduce the scanning cost and improve the patient experience. This can also potentially increase the image quality by reducing the motion artefacts and contrast washout. However, once an image field of view and the desired resolution are chosen, the minimum scanning time is normally determined by the requirement of acquiring sufficient raw data to meet the Nyquist-Shannon sampling criteria. Compressive Sensing (CS) theory has been perfectly matched to the MRI scanning sequence design with much l…
Robust Identity Perceptual Watermark Against Deepfake Face Swapping
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)（山东人工智能研究院，齐鲁工业大学（山东科学院））；School of Cyber Security, Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东科学院）网络安全学院）；Key Laboratory of Computing Power Network and Information Security, Ministry of Education, China（教育部计算能力网络与信息安全重点实验室）
ITU Vision Lab（伊斯坦布尔理工大学视觉实验室）；Istanbul Technical University（伊斯坦布尔理工大学）
Magnetic Resonance Angiography (MRA) has become an essential MR contrast for imaging and evaluation of vascular anatomy and related diseases. MRA acquisitions are typically ordered for vascular interventions, whereas in typical scenarios, MRA sequences can be absent in the patient scans. This motivates the need for a technique that generates inexistent MRA from existing MR multi-contrast, which could be a valuable tool in retrospective subject evaluations and imaging studies. In this paper, we present a generative adversarial network (GAN) based technique to generate MRA from T1-weighted and T…
FS-RWKV: Leveraging Frequency Spatial-Aware RWKV for 3T-to-7T MRI Translation
Faculty of Science and Technology, University of Macau（澳门大学科学与技术学院）；School of Electronic and Communication Engineering, Shenzhen Polytechnic University（深圳职业技术学院电子与通信工程学院）；Department of Cardiology, Guangdong Provincial People’s Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China（广东省人民医院心内科（广东省医学科学院））；Guangdong Cardiovascular Institute, Guangdong Provincial People’s Hospital, Guangdong Academy of Medical Sciences, Guangzhou, China（广东省心血管病研究院，广东省人民医院，广东省医学科学院，广州，中国）
ContourDiff: Unpaired Medical Image Translation with Structural Consistency
Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）；Department of Biostatistics & Bioinformatics, Duke University（杜克大学生物统计学与生物信息学系）；Department of Radiology, Duke University（杜克大学放射学系）；Department of Computer Science, Duke University（杜克大学计算机科学系）
Deep learning networks have shown promising results in fast magnetic resonance imaging (MRI) reconstruction. In our work, we develop deep networks to further improve the quantitative and the perceptual quality of reconstruction. To begin with, we propose reconsynergynet (RSN), a network that combines the complementary benefits of independently operating on both the image and the Fourier domain. For a single-coil acquisition, we introduce deep cascade RSN (DC-RSN), a cascade of RSN blocks interleaved with data fidelity (DF) units. Secondly, we improve the structure recovery of DC-RSN for T2 wei…
PathoSyn: Imaging-Pathology MRI Synthesis via Disentangled Deviation Diffusion
Department of Radiology, Boston Children’s Hospital, Harvard Medical School（放射科，波士顿儿童医院，哈佛医学院）；College of Science, Northeastern University（科学学院，东北大学）；School of Medicine, Yale University（医学院，耶鲁大学）；Department of Cardiac Surgery, The Second Affiliated Hospital of Jiangxi Medical College, Nanchang University（心脏外科科，江西医学院第二附属医院，南昌大学）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition…
Friedrich-Alexander-Universität（弗里德里希-亚历山大大学）；Imperial College（帝国理工学院）；National Heart & Lung Institute, Imperial College（帝国理工学院国家心肺研究所）；Royal Brompton Hospital（皇家布朗普顿医院）
Deep learning approaches have shown promising performance for compressed sensing-based Magnetic Resonance Imaging. While deep neural networks trained with mean squared error (MSE) loss functions can achieve high peak signal to noise ratio, the reconstructed images are often blurry and lack sharp details, especially for higher undersampling rates. Recently, adversarial and perceptual loss functions have been shown to achieve more visually appealing results. However, it remains an open question how to (1) optimally combine these loss functions with the MSE loss function and (2) evaluate such a p…
A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgery
Department of Neurosurgery, Neurovascular Unit Río Hortega University Hospital（里奥霍尔特ega大学医院神经外科部门，神经血管单元）；Specialized Group in Biomedical Imaging and Computational Analysis (GEIBAC)（生物医学成像与计算分析专项组（GEIBAC））；Instituto de Investigación Biosanitaria de Valladolid (IBioVALL)（瓦尔拉多利德生物医学研究 institute（IBioVALL））
Removing Motion Artifact in MRI by Using a Perceptual Loss Driven Deep Learning Framework
Institute of Magnetic Resonance and Molecular Imaging in Medicine（医学磁共振与分子成像研究所）；Shanghai Key Laboratory of Magnetic Resonance（上海磁共振重点实验室）；The First Affiliated Hospital of Naval Medical University Department of Radiology（海军医学院第一附属医院放射科）；Laboratory for Intelligent Medical Imaging（智能医学成像实验室）；Tsinghua Cross-Strait Research Institute（清华大学海峡研究院）；RadioDynamic Medical（RadioDynamic医疗）
Cyclic 2.5D Perceptual Loss for Cross-Modal 3D Medical Image Synthesis: T1w MRI to Tau PET
Department of Artificial Intelligence, Hanyang University, Seoul, South Korea（人工智能系，翰阳大学，首尔，韩国）；Department of Electronic Engineering, Hanyang University, Seoul, South Korea（电子工程系，翰阳大学，首尔，韩国）；Division of Computer Engineering, Hankuk University of Foreign Studies, Yongin, South Korea（计算机工程系，韩国外语大学， Yongin，韩国）；Division of AI Data Convergence, Hankuk University of Foreign Studies, Yongin, South Korea（AI数据融合系，韩国外语大学， Yongin，韩国）；Division of Language & AI, Hankuk University of Foreign Studies, Seoul, South Korea（语言与AI系，韩国外语大学，首尔，韩国）
Recovering images from undersampled linear measurements typically leads to an ill-posed linear inverse problem, that asks for proper statistical priors. Building effective priors is however challenged by the low train and test overhead dictated by real-time tasks; and the need for retrieving visually "plausible" and physically "feasible" images with minimal hallucination. To cope with these challenges, we design a cascaded network architecture that unrolls the proximal gradient iterations by permeating benefits from generative residual networks (ResNet) to modeling the proximal operator. A mix…
Sichuan University（四川大学）；Chengdu University of Information Technology（成都信息工程大学）；Southeast University（东南大学）；West China Hospital of Sichuan University（四川大学华西医院）；Southern Medical University（南方医科大学）
Structure-preserved denoising of 3D magnetic resonance imaging (MRI) images is a critical step in medical image analysis. Over the past few years, many algorithms with impressive performances have been proposed. In this paper, inspired by the idea of deep learning, we introduce an MRI denoising method based on the residual encoder-decoder Wasserstein generative adversarial network (RED-WGAN). Specifically, to explore the structure similarity between neighboring slices, a 3D configuration is utilized as the basic processing unit. Residual autoencoders combined with deconvolution operations are …
Fudan University（复旦大学）；University of Maryland（马里兰大学）
Online media data, in the forms of images and videos, are becoming mainstream communication channels. However, recent advances in deep learning, particularly deep generative models, open the doors for producing perceptually convincing images and videos at a low cost, which not only poses a serious threat to the trustworthiness of digital information but also has severe societal implications. This motivates a growing interest of research in media tampering detection, i.e., using deep learning techniques to examine whether media data have been maliciously manipulated. Depending on the content of…
Towards Robust Defense against Customization via Protective Perturbation Resistant to Diffusion-based Purification
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences（自动化研究所、中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院、中国科学院大学）
a Division of Medical Informatics, Department of Biomedical Engineering, Link\" o ping University, Link\" o ping, Sweden b Division of Statistics；Machine Learning, Department of Computer；Information Science, Link\" o ping University, Link\" o ping, Sweden c Center for Medical Image Science；Visualization (CMIV), Link\" o ping University, Link\" o ping, Sweden
In medical imaging, a general problem is that it is costly and time consuming to collect high quality data from healthy and diseased subjects. Generative adversarial networks (GANs) is a deep learning method that has been developed for synthesizing data. GANs can thereby be used to generate more realistic training data, to improve classification performance of machine learning algorithms. Another application of GANs is image-to-image translations, e.g. generating magnetic resonance (MR) images from computed tomography (CT) images, which can be used to obtain multimodal datasets from a single m…
Technische Universität München（慕尼黑工业大学）；Harvard University（哈佛大学）；Massachusetts General Hospital（麻省总医院）
Undersampling the k-space data is widely adopted for acceleration of Magnetic Resonance Imaging (MRI). Current deep learning based approaches for supervised learning of MRI image reconstruction employ real-valued operations and representations by treating complex valued k-space/spatial-space as real values. In this paper, we propose complex dense fully convolutional neural network ($\mathbb{C}$DFNet) for learning to de-alias the reconstruction artifacts within undersampled MRI images. We fashioned a densely-connected fully convolutional block tailored for complex-valued inputs by introducing d…
On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital（麻省总医院Athinoula A. Martinos生物医学成像中心）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）
Generative Drifting for Conditional Medical Image Generation
Department of Orthodontics and Orofacial Orthopedics, Friedrich-Alexander-University Erlangen-Nuremberg（口腔正畸与面部骨科系，弗里德里希-艾萨克-埃尔兰-纽伦堡大学）；Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-University Erlangen-Nuremberg（生物医学工程人工智能系，弗里德里希-艾萨克-埃尔兰-纽伦堡大学）；Pattern Recognition Lab, Friedrich-Alexander-University Erlangen-Nuremberg（模式识别实验室，弗里德里希-艾萨克-埃尔兰-纽伦堡大学）；Department of Biomedical Engineering, Sun-Yat-sen University（生物医学工程系，孙中山大学）
