Phase retrieval, or the process of recovering phase information in reciprocal space to reconstruct images from measured intensity alone, is the underlying basis to a variety of imaging applications including coherent diffraction imaging (CDI). Typical phase retrieval algorithms are iterative in nature, and hence, are time-consuming and computationally expensive, precluding real-time imaging. Furthermore, iterative phase retrieval algorithms struggle to converge to the correct solution especially in the presence of strong phase structures. In this work, we demonstrate the training and testing of CDI NN, a pair of deep deconvolutional networks trained to predict structure and phase in real space of a 2D object from its corresponding far-field diffraction intensities alone. Once trained, CDI NN can invert a diffraction pattern to an image within a few milliseconds of compute time on a standard desktop machine, opening the door to real-time imaging.
Khoury College of Computer Sciences, Northeastern University（东北大学库理计算机学院）；Department of Mathematics, Northeastern University（东北大学数学系）
Generative Adversarial Networks (GANs) have been shown to be powerful and flexible priors when solving inverse problems. One challenge of using them is overcoming representation error, the fundamental limitation of the network in representing any particular signal. Recently, multiple proposed inversion algorithms reduce representation error by optimizing over intermediate layer representations. These methods are typically applied to generative models that were trained agnostic of the downstream inversion algorithm. In our work, we introduce a principle that if a generative model is intended fo…
We present an application of deep generative models in the context of partial-differential equation (PDE) constrained inverse problems. We combine a generative adversarial network (GAN) representing an a priori model that creates subsurface geological structures and their petrophysical properties, with the numerical solution of the PDE governing the propagation of acoustic waves within the earth's interior. We perform Bayesian inversion using an approximate Metropolis-adjusted Langevin algorithm (MALA) to sample from the posterior given seismic observations. Gradients with respect to the model…
DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models
Rutgers University（新泽西罗格斯大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；Red Hat AI Innovation（红帽AI创新）；Google DeepMind（谷歌DeepMind）；NYU（纽约大学）；Walmart Global Tech（沃尔玛全球技术）；NEC Labs America（NEC美国实验室）；Massachusetts Institute of Technology（麻省理工学院）；ANU（澳大利亚国立大学）；UT Arlington（德克萨斯大学阿灵顿分校）
Imperial College London（帝国理工学院）；Open Energi Ltd.（Open Energi有限公司）；Technical University of Denmark（丹麦技术大学）；Euclidity SL（Euclidity SL公司）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）；Expero Inc.（Expero公司）
Traditional physics-based approaches to infer sub-surface properties such as full-waveform inversion or reflectivity inversion are time-consuming and computationally expensive. We present a deep-learning technique that eliminates the need for these computationally complex methods by posing the problem as one of domain transfer. Our solution is based on a deep convolutional generative adversarial network and dramatically reduces computation time. Training based on two different types of synthetic data produced a neural network that generates realistic velocity models when applied to a real data…
Deep generative models (e.g. GANs and VAEs) have been developed quite extensively in recent years. Lately, there has been an increased interest in the inversion of such a model, i.e. given a (possibly corrupted) signal, we wish to recover the latent vector that generated it. Building upon sparse representation theory, we define conditions that are applicable to any inversion algorithm (gradient descent, deep encoder, etc.), under which such generative models are invertible with a unique solution. Importantly, the proposed analysis is applicable to any trained model, and does not depend on Gaus…
Most adversarial attack defense methods rely on obfuscating gradients. These methods are successful in defending against gradient-based attacks; however, they are easily circumvented by attacks which either do not use the gradient or by attacks which approximate and use the corrected gradient. Defenses that do not obfuscate gradients such as adversarial training exist, but these approaches generally make assumptions about the attack such as its magnitude. We propose a classification model that does not obfuscate gradients and is robust by construction without assuming prior knowledge about the…
Mobile edge devices see increased demands in deep neural networks (DNNs) inference while suffering from stringent constraints in computing resources. Split computing (SC) emerges as a popular approach to the issue by executing only initial layers on devices and offloading the remaining to the cloud. Prior works usually assume that SC offers privacy benefits as only intermediate features, instead of private data, are shared from devices to the cloud. In this work, we debunk this SC-induced privacy protection by (i) presenting a novel data-free model inversion method and (ii) demonstrating sampl…
TAG-WM: Tamper-Aware Generative Image Watermarking via Diffusion Inversion Sensitivity
Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China（安徽省数字安全重点实验室，中国科学技术大学）；School of Computing, National University of Singapore（computing学院，新加坡国立大学）
Belgian Nuclear Research Centre（比利时核研究中心）；Normandie Univ（诺曼底大学）；UNIROUEN（鲁昂大学）；UNIHAVRE（勒阿弗尔大学）；INSA Rouen（鲁昂国立应用科学学院）；LITIS（鲁昂信息科学与技术实验室）；University of Lausanne（洛桑大学）
Probabilistic inversion within a multiple-point statistics framework is often computationally prohibitive for high-dimensional problems. To partly address this, we introduce and evaluate a new training-image based inversion approach for complex geologic media. Our approach relies on a deep neural network of the generative adversarial network (GAN) type. After training using a training image (TI), our proposed spatial GAN (SGAN) can quickly generate 2D and 3D unconditional realizations. A key characteristic of our SGAN is that it defines a (very) low-dimensional parameterization, thereby allowi…
University of Maryland（马里兰大学）；Butterfly Network（蝴蝶网络公司）
Inferring the latent variable generating a given test sample is a challenging problem in Generative Adversarial Networks (GANs). In this paper, we propose InvGAN - a novel framework for solving the inference problem in GANs, which involves training an encoder network capable of inverting a pre-trained generator network without access to any training data. Under mild assumptions, we theoretically show that using InvGAN, we can approximately invert the generations of any latent code of a trained GAN model. Furthermore, we empirically demonstrate the superiority of our inference scheme by quantit…
University of Science and Technology of China（中国科学技术大学）；University of Waterloo（滑铁卢大学）；CISPA Helmholtz Center for Information Security（CISPA赫尔姆霍茨信息安全中心）
Model inversion (MI) attacks have raised increasing concerns about privacy, which can reconstruct training data from public models. Indeed, MI attacks can be formalized as an optimization problem that seeks private data in a certain space. Recent MI attacks leverage a generative adversarial network (GAN) as an image prior to narrow the search space, and can successfully reconstruct even the high-dimensional data (e.g., face images). However, these generative MI attacks do not fully exploit the potential capabilities of the target model, still leading to a vague and coupled search space, i.e., …
Harbin Institute of Technology（哈尔滨工业大学）；Peking University（北京大学）
Seismic full waveform inversion (FWI) is a powerful geophysical imaging technique that produces high-resolution subsurface models by iteratively minimizing the misfit between the simulated and observed seismograms. Unfortunately, conventional FWI with least-squares function suffers from many drawbacks such as the local-minima problem and computation of explicit gradient. It is particularly challenging with the contaminated measurements or poor starting models. Recent works relying on partial differential equations and neural networks show promising performance for two-dimensional FWI. Inspired…
Beckman Institute for Advanced Science and Technology（贝克曼先进科学技术研究所）；Cancer Center at Illinois（伊利诺伊癌症中心）
Infrared (IR) microscopes measure spectral information that quantifies molecular content to assign the identity of biomedical cells but lack the spatial quality of optical microscopy to appreciate morphologic features. Here, we propose a method to utilize the semantic information of cellular identity from IR imaging with the morphologic detail of pathology images in a deep learning-based approach to image super-resolution. Using Generative Adversarial Networks (GANs), we enhance the spatial detail in IR imaging beyond the diffraction limit while retaining their spectral contrast. This techniqu…
Generative adversarial networks (GANs) learn a deep generative model that is able to synthesise novel, high-dimensional data samples. New data samples are synthesised by passing latent samples, drawn from a chosen prior distribution, through the generative model. Once trained, the latent space exhibits interesting properties, that may be useful for down stream tasks such as classification or retrieval. Unfortunately, GANs do not offer an "inverse model", a mapping from data space back to latent space, making it difficult to infer a latent representation for a given data sample. In this paper, …
Generative adversarial networks (GANs) have shown remarkable success in generation of unstructured data, such as, natural images. However, discovery and separation of modes in the generated space, essential for several tasks beyond naive data generation, is still a challenge. In this paper, we address the problem of imposing desired modal properties on the generated space using a latent distribution, engineered in accordance with the modal properties of the true data distribution. This is achieved by training a latent space inversion network in tandem with the generative network using a diverg…
Columbia University（哥伦比亚大学）；Rutgers University（罗格斯大学）
Many machine learning methods operate by inverting a neural network at inference time, which has become a popular technique for solving inverse problems in computer vision, robotics, and graphics. However, these methods often involve gradient descent through a highly non-convex loss landscape, causing the optimization process to be unstable and slow. We introduce a method that learns a loss landscape where gradient descent is efficient, bringing massive improvement and acceleration to the inversion process. We demonstrate this advantage on a number of methods for both generative and discrimina…
TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；City University of Hong Kong（香港城市大学）；Shenzhen Loop Area Institute（深圳河套学院）；Peng Cheng Laboratory（鹏城实验室）；Shandong University（山东大学）
HKUST（香港科技大学）；SenseTime Research（商汤科技研究院）；Centre for Artificial Intelligence and Robotics（人工智能与机器人中心）
We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views while preserving specific details of the input image. High-fidelity 3D GAN inversion is inherently challenging due to the geometry-texture trade-off in 3D inversion, where overfitting to a single view input image often damages the estimated geometry during the latent optimization. To solve this challenge, we propose a novel pipeline that builds on the pseudo-multi-view estimation with visibility analysis. We keep the original textures for the visible parts and …
Czech Technical University in Prague（布拉格捷克技术大学）；Faculty of Electrical Engineering（电气工程学院）；Adobe Research（奥多比研究院）
In this short report, we present a simple, yet effective approach to editing real images via generative adversarial networks (GAN). Unlike previous techniques, that treat all editing tasks as an operation that affects pixel values in the entire image in our approach we cut up the image into a set of smaller segments. For those segments corresponding latent codes of a generative network can be estimated with greater accuracy due to the lower number of constraints. When codes are altered by the user the content in the image is manipulated locally while the rest of it remains unaffected. Thanks t…
GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）
School of Computer Science and Engineering（计算机科学与工程学院）；Facebook AI Research（脸书人工智能研究院）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
One of the main motivations for training high quality image generative models is their potential use as tools for image manipulation. Recently, generative adversarial networks (GANs) have been able to generate images of remarkable quality. Unfortunately, adversarially-trained unconditional generator networks have not been successful as image priors. One of the main requirements for a network to act as a generative image prior, is being able to generate every possible image from the target distribution. Adversarial learning often experiences mode-collapse, which manifests in generators that can…
Image reconstruction under multiple light scattering is crucial in a number of applications such as diffraction tomography. The reconstruction problem is often formulated as a nonconvex optimization, where a nonlinear measurement model is used to account for multiple scattering and regularization is used to enforce prior constraints on the object. In this paper, we propose a powerful alternative to this optimization-based view of image reconstruction by designing and training a deep convolutional neural network that can invert multiple scattered measurements to produce a high-quality image of …
Image Inversion: A Survey from GANs to Diffusion and Beyond
College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；YouTu Lab, Tencent（腾讯优图实验室）；College of Computer and Information Science, Southwest University（西南大学计算机与信息科学学院）；Department of Computer Science & Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Hong Kong University of Science and Technology（香港科技大学）；VinAI Research（VinAI研究院）；Deakin University（迪肯大学）
In this paper, we propose a new method for mapping a 3D point cloud to the latent space of a 3D generative adversarial network. Our generative model for 3D point clouds is based on SP-GAN, a state-of-the-art sphere-guided 3D point cloud generator. We derive an efficient way to encode an input 3D point cloud to the latent space of the SP-GAN. Our point cloud encoder can resolve the point ordering issue during inversion, and thus can determine the correspondences between points in the generated 3D point cloud and those in the canonical sphere used by the generator. We show that our method outper…
