Multi-Modality Image Inpainting using Generative Adversarial Networks

Inpainting borehole images using Generative Adversarial Networks

In this paper, we propose a GAN-based approach for gap filling in borehole images created by wireline microresistivity imaging tools. The proposed method utilizes a generator, global discriminator, and local discriminator to inpaint the missing regions of the image. The generator is based on an auto-encoder architecture with skip-connections, and the loss function used is the Wasserstein GAN loss. Our experiments on a dataset of borehole images demonstrate that the proposed model can effectively deal with large-scale missing pixels and generate realistic completion results. This approach can i…

RGB-D Image Inpainting Using Generative Adversarial Network with a Late Fusion Approach

Diminished reality is a technology that aims to remove objects from video images and fills in the missing region with plausible pixels. Most conventional methods utilize the different cameras that capture the same scene from different viewpoints to allow regions to be removed and restored. In this paper, we propose an RGB-D image inpainting method using generative adversarial network, which does not require multiple cameras. Recently, an RGB image inpainting method has achieved outstanding results by employing a generative adversarial network. However, RGB inpainting methods aim to restore onl…

Patch-Based Image Inpainting with Generative Adversarial Networks

Area of image inpainting over relatively large missing regions recently advanced substantially through adaptation of dedicated deep neural networks. However, current network solutions still introduce undesired artifacts and noise to the repaired regions. We present an image inpainting method that is based on the celebrated generative adversarial network (GAN) framework. The proposed PGGAN method includes a discriminator network that combines a global GAN (G-GAN) architecture with a patchGAN approach. PGGAN first shares network layers between G-GAN and patchGAN, then splits paths to produce two…

WaveFill: A Wavelet-based Generation Network for Image Inpainting

Image inpainting aims to complete the missing or corrupted regions of images with realistic contents. The prevalent approaches adopt a hybrid objective of reconstruction and perceptual quality by using generative adversarial networks. However, the reconstruction loss and adversarial loss focus on synthesizing contents of different frequencies and simply applying them together often leads to inter-frequency conflicts and compromised inpainting. This paper presents WaveFill, a wavelet-based inpainting network that decomposes images into multiple frequency bands and fills the missing regions in e…

基于分割混淆对抗训练与对比学习的生成式图像修复

Generative Image Inpainting with Segmentation Confusion Adversarial Training and Contrastive Learning

本文提出融合SCAT、全局对抗训练及纹理与语义对比损失的图像修复框架，兼顾全局一致性、局部纹理和自由孔洞适应性，并在两个基准上验证其优越性。

PiiGAN: Generative Adversarial Networks for Pluralistic Image Inpainting

The latest methods based on deep learning have achieved amazing results regarding the complex work of inpainting large missing areas in an image. But this type of method generally attempts to generate one single "optimal" result, ignoring many other plausible results. Considering the uncertainty of the inpainting task, one sole result can hardly be regarded as a desired regeneration of the missing area. In view of this weakness, which is related to the design of the previous algorithms, we propose a novel deep generative model equipped with a brand new style extractor which can extract the sty…

四元数生成对抗神经网络及其在彩色图像修复中的应用

Quaternion Generative Adversarial Neural Networks and Applications to Color Image Inpainting

针对彩色图像通道相关性被忽视的问题，提出融合四元数反卷积与四元数批量归一化的QGAN，用于大面积缺失彩色图像修复并取得优于现有算法的效果。

Unsupervised Region-based Anomaly Detection in Brain MRI with Adversarial Image Inpainting

Medical segmentation is performed to determine the bounds of regions of interest (ROI) prior to surgery. By allowing the study of growth, structure, and behaviour of the ROI in the planning phase, critical information can be obtained, increasing the likelihood of a successful operation. Usually, segmentations are performed manually or via machine learning methods trained on manual annotations. In contrast, this paper proposes a fully automatic, unsupervised inpainting-based brain tumour segmentation system for T1-weighted MRI. First, a deep convolutional neural network (DCNN) is trained to rec…

对抗性掩码图像修复用于猴痘与非猴痘的稳健检测

Adversarial Masked Image Inpainting for Robust Detection of Mpox and Non-Mpox

针对现有分类模型易受噪声干扰、依赖多样非猴痘图像且无法检测异常输入的缺点，提出MIM方法，仅用生成对抗网络修复猴痘图像并度量相似性来检测，平均AUROC达0.8237，并开发了免费手机应用。

Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart

Recent advances in deep generative models have shown promising potential in image inpanting, which refers to the task of predicting missing pixel values of an incomplete image using the known context. However, existing methods can be slow or generate unsatisfying results with easily detectable flaws. In addition, there is often perceivable discontinuity near the holes and require further post-processing to blend the results. We present a new approach to address the difficulty of training a very deep generative model to synthesize high-quality photo-realistic inpainting. Our model uses conditio…

R-MNet: A Perceptual Adversarial Network for Image Inpainting

Facial image inpainting is a problem that is widely studied, and in recent years the introduction of Generative Adversarial Networks, has led to improvements in the field. Unfortunately some issues persists, in particular when blending the missing pixels with the visible ones. We address the problem by proposing a Wasserstein GAN combined with a new reverse mask operator, namely Reverse Masking Network (R-MNet), a perceptual adversarial network for image inpainting. The reverse mask operator transfers the reverse masked image to the end of the encoder-decoder network leaving only valid pixels …

EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning

Over the last few years, deep learning techniques have yielded significant improvements in image inpainting. However, many of these techniques fail to reconstruct reasonable structures as they are commonly over-smoothed and/or blurry. This paper develops a new approach for image inpainting that does a better job of reproducing filled regions exhibiting fine details. We propose a two-stage adversarial model EdgeConnect that comprises of an edge generator followed by an image completion network. The edge generator hallucinates edges of the missing region (both regular and irregular) of the image…

DAFT-GAN：用于文本引导图像修复的双仿射变换生成对抗网络

DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting

针对文本引导图像修复中的图文对齐与区域分布一致性问题，提出DAFT-GAN，通过双仿射变换逐步融合特征并分区编码以减少信息泄漏，在三个基准上取得优于现有GAN模型的效果。

A Wasserstein GAN for Joint Learning of Inpainting and Spatial Optimisation

Image inpainting is a restoration method that reconstructs missing image parts. However, a carefully selected mask of known pixels that yield a high quality inpainting can also act as a sparse image representation. This challenging spatial optimisation problem is essential for practical applications such as compression. So far, it has been almost exclusively adressed by model-based approaches. First attempts with neural networks seem promising, but are tailored towards specific inpainting operators or require postprocessing. To address this issue, we propose the first generative adversarial ne…

Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network

Contemporary benchmark methods for image inpainting are based on deep generative models and specifically leverage adversarial loss for yielding realistic reconstructions. However, these models cannot be directly applied on image/video sequences because of an intrinsic drawback- the reconstructions might be independently realistic, but, when visualized as a sequence, often lacks fidelity to the original uncorrupted sequence. The fundamental reason is that these methods try to find the best matching latent space representation near to natural image manifold without any explicit distance based lo…

Semantic Image Inpainting Through Improved Wasserstein Generative Adversarial Networks

Image inpainting is the task of filling-in missing regions of a damaged or incomplete image. In this work we tackle this problem not only by using the available visual data but also by incorporating image semantics through the use of generative models. Our contribution is twofold: First, we learn a data latent space by training an improved version of the Wasserstein generative adversarial network, for which we incorporate a new generator and discriminator architecture. Second, the learned semantic information is combined with a new optimization loss for inpainting whose minimization infers the…

Vision-Infused Deep Audio Inpainting

Multi-modality perception is essential to develop interactive intelligence. In this work, we consider a new task of visual information-infused audio inpainting, \ie synthesizing missing audio segments that correspond to their accompanying videos. We identify two key aspects for a successful inpainter: (1) It is desirable to operate on spectrograms instead of raw audios. Recent advances in deep semantic image inpainting could be leveraged to go beyond the limitations of traditional audio inpainting. (2) To synthesize visually indicated audio, a visual-audio joint feature space needs to be learn…

AdvPaint：通过对抗性注意力破坏保护图像免受修复篡改

AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption

针对扩散模型被滥用于未授权图像修复的问题，本文提出 ADVPAINT，通过攻击自注意力与交叉注意力并采用两阶段扰动，显著提升防御效果。

Face Completion with Semantic Knowledge and Collaborative Adversarial Learning

Unlike a conventional background inpainting approach that infers a missing area from image patches similar to the background, face completion requires semantic knowledge about the target object for realistic outputs. Current image inpainting approaches utilize generative adversarial networks (GANs) to achieve such semantic understanding. However, in adversarial learning, the semantic knowledge is learned implicitly and hence good semantic understanding is not always guaranteed. In this work, we propose a collaborative adversarial learning approach to face completion to explicitly induce the tr…

Audio inpainting with generative adversarial network

We study the ability of Wasserstein Generative Adversarial Network (WGAN) to generate missing audio content which is, in context, (statistically similar) to the sound and the neighboring borders. We deal with the challenge of audio inpainting long range gaps (500 ms) using WGAN models. We improved the quality of the inpainting part using a new proposed WGAN architecture that uses a short-range and a long-range neighboring borders compared to the classical WGAN model. The performance was compared with two different audio instruments (piano and guitar) and on virtuoso pianists together with a st…

基于学习的生成对抗网络WiFi指纹修复

Learning-Based WiFi Fingerprint Inpainting via Generative Adversarial Networks

本研究提出基于生成对抗网络的WiFi指纹修复方法，解决室内定位中指纹收集不完整问题，通过捕获AP间相关性并引入对抗判别器提升修复质量。

Multi-Modality Image Super-Resolution using Generative Adversarial Networks

Over the past few years deep learning-based techniques such as Generative Adversarial Networks (GANs) have significantly improved solutions to image super-resolution and image-to-image translation problems. In this paper, we propose a solution to the joint problem of image super-resolution and multi-modality image-to-image translation. The problem can be stated as the recovery of a high-resolution image in a modality, given a low-resolution observation of the same image in an alternative modality. Our paper offers two models to address this problem and will be evaluated on the recovery of high…

通过潜在空间优化实现一致性和多模态图像修复

Coherent and Multi-modality Image Inpainting via Latent Space Optimization

PILOT通过潜在空间优化提升图像修复的一致性和多模态处理能力。

Adversarially robust segmentation models learn perceptually-aligned gradients

The effects of adversarial training on semantic segmentation networks has not been thoroughly explored. While previous work has shown that adversarially-trained image classifiers can be used to perform image synthesis, we have yet to understand how best to leverage an adversarially-trained segmentation network to do the same. Using a simple optimizer, we demonstrate that adversarially-trained semantic segmentation networks can be used to perform image inpainting and generation. Our experiments demonstrate that adversarially-trained segmentation networks are more robust and indeed exhibit perce…

Inpainting via Generative Adversarial Networks for CMB data analysis

In this work, we propose a new method to inpaint the CMB signal in regions masked out following a point source extraction process. We adopt a modified Generative Adversarial Network (GAN) and compare different combinations of internal (hyper-)parameters and training strategies. We study the performance using a suitable $\mathcal{C}_r$ variable in order to estimate the performance regarding the CMB power spectrum recovery. We consider a test set where one point source is masked out in each sky patch with a 1.83 $\times$ 1.83 squared degree extension, which, in our gridding, corresponds to 64 $\…

Shape Inpainting using 3D Generative Adversarial Network and Recurrent Convolutional Networks

Recent advances in convolutional neural networks have shown promising results in 3D shape completion. But due to GPU memory limitations, these methods can only produce low-resolution outputs. To inpaint 3D models with semantic plausibility and contextual details, we introduce a hybrid framework that combines a 3D Encoder-Decoder Generative Adversarial Network (3D-ED-GAN) and a Long-term Recurrent Convolutional Network (LRCN). The 3D-ED-GAN is a 3D convolutional neural network trained with a generative adversarial paradigm to fill missing 3D data in low-resolution. LRCN adopts a recurrent neura…

V-LinkNet: Learning Contextual Inpainting Across Latent Space of Generative Adversarial Network

Image inpainting is a key technique in image processing task to predict the missing regions and generate realistic images. Given the advancement of existing generative inpainting models with feature extraction, propagation and reconstruction capabilities, there is lack of high-quality feature extraction and transfer mechanisms in deeper layers to tackle persistent aberrations on the generated inpainted regions. Our method, V-LinkNet, develops high-level feature transference to deep level textural context of inpainted regions our work, proposes a novel technique of combining encoders learning t…

Context-Aware Semantic Inpainting

Recently image inpainting has witnessed rapid progress due to generative adversarial networks (GAN) that are able to synthesize realistic contents. However, most existing GAN-based methods for semantic inpainting apply an auto-encoder architecture with a fully connected layer, which cannot accurately maintain spatial information. In addition, the discriminator in existing GANs struggle to understand high-level semantics within the image context and yield semantically consistent content. Existing evaluation criteria are biased towards blurry results and cannot well characterize edge preservatio…

利用半超分辨率GAN对损坏图像进行图像修复

Image inpainting for corrupted images by using the semi-super resolution GAN

本文提出了一种半超分辨率GAN模型，用于修复受损图像，通过不同水平的像素损坏训练以提高图像修复的准确性和质量。

Fashion Editing with Adversarial Parsing Learning

Interactive fashion image manipulation, which enables users to edit images with sketches and color strokes, is an interesting research problem with great application value. Existing works often treat it as a general inpainting task and do not fully leverage the semantic structural information in fashion images. Moreover, they directly utilize conventional convolution and normalization layers to restore the incomplete image, which tends to wash away the sketch and color information. In this paper, we propose a novel Fashion Editing Generative Adversarial Network (FE-GAN), which is capable of ma…

Pixel-wise Dense Detector for Image Inpainting

Recent GAN-based image inpainting approaches adopt an average strategy to discriminate the generated image and output a scalar, which inevitably lose the position information of visual artifacts. Moreover, the adversarial loss and reconstruction loss (e.g., l1 loss) are combined with tradeoff weights, which are also difficult to tune. In this paper, we propose a novel detection-based generative framework for image inpainting, which adopts the min-max strategy in an adversarial process. The generator follows an encoder-decoder architecture to fill the missing regions, and the detector using wea…

fastWDM3D：快速且精确的3D健康组织修复

fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting

本文提出fastWDM3D，通过将方差保持噪声调度与重建损失应用于无GAN的3D小波扩散模型，在仅两个时间步内实现快速精确的3D健康脑组织修复，速度较其他DDPM提升最高800倍且性能更优。

Region-wise Generative Adversarial ImageInpainting for Large Missing Areas

Recently deep neutral networks have achieved promising performance for filling large missing regions in image inpainting tasks. They usually adopted the standard convolutional architecture over the corrupted image, leading to meaningless contents, such as color discrepancy, blur and artifacts. Moreover, most inpainting approaches cannot well handle the large continuous missing area cases. To address these problems, we propose a generic inpainting framework capable of handling with incomplete images on both continuous and discontinuous large missing areas, in an adversarial manner. From which, …

Boundless: Generative Adversarial Networks for Image Extension

Image extension models have broad applications in image editing, computational photography and computer graphics. While image inpainting has been extensively studied in the literature, it is challenging to directly apply the state-of-the-art inpainting methods to image extension as they tend to generate blurry or repetitive pixels with inconsistent semantics. We introduce semantic conditioning to the discriminator of a generative adversarial network (GAN), and achieve strong results on image extension with coherent semantics and visually pleasing colors and textures. We also show promising res…

DAM-GAN : Image Inpainting using Dynamic Attention Map based on Fake Texture Detection

Deep neural advancements have recently brought remarkable image synthesis performance to the field of image inpainting. The adaptation of generative adversarial networks (GAN) in particular has accelerated significant progress in high-quality image reconstruction. However, although many notable GAN-based networks have been proposed for image inpainting, still pixel artifacts or color inconsistency occur in synthesized images during the generation process, which are usually called fake textures. To reduce pixel inconsistency disorder resulted from fake textures, we introduce a GAN-based model u…

PIINET: A 360-degree Panoramic Image Inpainting Network Using a Cube Map

Inpainting has been continuously studied in the field of computer vision. As artificial intelligence technology developed, deep learning technology was introduced in inpainting research, helping to improve performance. Currently, the input target of an inpainting algorithm using deep learning has been studied from a single image to a video. However, deep learning-based inpainting technology for panoramic images has not been actively studied. We propose a 360-degree panoramic image inpainting method using generative adversarial networks (GANs). The proposed network inputs a 360-degree equirecta…

Continuation of Famous Art with AI: A Conditional Adversarial Network Inpainting Approach

Much of the state-of-the-art in image synthesis inspired by real artwork are either entirely generative by filtered random noise or inspired by the transfer of style. This work explores the application of image inpainting to continue famous artworks and produce generative art with a Conditional GAN. During the training stage of the process, the borders of images are cropped, leaving only the centre. An inpainting GAN is then tasked with learning to reconstruct the original image from the centre crop by way of minimising both adversarial and absolute difference losses, which are analysed by bot…

Faster Unsupervised Semantic Inpainting: A GAN Based Approach

In this paper, we propose to improve the inference speed and visual quality of contemporary baseline of Generative Adversarial Networks (GAN) based unsupervised semantic inpainting. This is made possible with better initialization of the core iterative optimization involved in the framework. To our best knowledge, this is also the first attempt of GAN based video inpainting with consideration to temporal cues. On single image inpainting, we achieve about 4.5-5$\times$ speedup and 80$\times$ on videos compared to baseline. Simultaneously, our method has better spatial and temporal reconstructio…

Multi-Modality Generative Adversarial Networks with Tumor Consistency Loss for Brain MR Image Synthesis

Magnetic Resonance (MR) images of different modalities can provide complementary information for clinical diagnosis, but whole modalities are often costly to access. Most existing methods only focus on synthesizing missing images between two modalities, which limits their robustness and efficiency when multiple modalities are missing. To address this problem, we propose a multi-modality generative adversarial network (MGAN) to synthesize three high-quality MR modalities (FLAIR, T1 and T1ce) from one MR modality T2 simultaneously. The experimental results show that the quality of the synthesize…

GRIG：少样本生成式残差图像修复

GRIG: Few-Shot Generative Residual Image Inpainting

针对图像修复依赖大规模训练数据的问题，本文提出GRIG少样本生成式残差方法，结合CNN、Transformer、双尺度判别器与伪造块对抗训练，实现高质量修复并在定量和定性上超越既有方法。

WavePaint：用于自监督修复的资源高效型Token混合器

WavePaint: Resource-efficient Token-mixer for Self-supervised Inpainting

本文提出基于WaveMix的全卷积架构WavePaint，利用二维离散小波变换进行token混合，在重建质量和计算效率上超越现有图像修复模型。

生成对抗网络用于脑图像合成：综述

Generative Adversarial Networks for Brain Images Synthesis: A Review

本文综述了生成对抗网络（GAN）在脑图像跨模态合成中的最新进展，涵盖CT、PET和MRI之间的相互转换，以解决多模态影像获取成本高、耗时的问题。

安全交通标志识别：一种抵御光补丁攻击的、启用注意力机制的通用图像修复机制

Secure Traffic Sign Recognition: An Attention-Enabled Universal Image Inpainting Mechanism against Light Patch Attacks

针对对抗性光补丁攻击交通标志识别的安全威胁，本文提出 SafeSign，通过二值掩码 U-Net 生成训练数据并结合注意力多视图融合修复污染图像，使三类识别模型平均准确率提升54.8%。

PEPSI++: Fast and Lightweight Network for Image Inpainting

Among the various generative adversarial network (GAN)-based image inpainting methods, a coarse-to-fine network with a contextual attention module (CAM) has shown remarkable performance. However, owing to two stacked generative networks, the coarse-to-fine network needs numerous computational resources such as convolution operations and network parameters, which result in low speed. To address this problem, we propose a novel network architecture called PEPSI: parallel extended-decoder path for semantic inpainting network, which aims at reducing the hardware costs and improving the inpainting …

基于潜在扩散的腰椎MRI病理图像修复

Inpainting Pathology in Lumbar Spine MRI with Latent Diffusion

提出基于潜在扩散模型和体素级噪声调度的MRI病理图像修复方法，有效解决数据不平衡问题，在腰椎T2 MRI上实现更优的病理特征插入效果。

基于扩散模型的多任务脑肿瘤图像修复：一份方法学报告

Multitask Brain Tumor Inpainting with Diffusion Models: A Methodological Report

针对医学数据集稀缺影响深度学习性能的问题，提出基于DDPM的多任务脑肿瘤图像修复方法，在多种脑MRI序列上生成高质量且多样的合成数据，并提供了概念验证与开源接口。

Sinogram Enhancement with Generative Adversarial Networks using Shape Priors

Compensating scarce measurements by inferring them from computational models is a way to address ill-posed inverse problems. We tackle Limited Angle Tomography by completing the set of acquisitions using a generative model and prior-knowledge about the scanned object. Using a Generative Adversarial Network as model and Computer-Assisted Design data as shape prior, we demonstrate a quantitative and qualitative advantage of our technique over other state-of-the-art methods. Inferring a substantial number of consecutive missing measurements, we offer an alternative to other image inpainting techn…

Physics-informed semantic inpainting: Application to geostatistical modeling

A fundamental problem in geostatistical modeling is to infer the heterogeneous geological field based on limited measurements and some prior spatial statistics. Semantic inpainting, a technique for image processing using deep generative models, has been recently applied for this purpose, demonstrating its effectiveness in dealing with complex spatial patterns. However, the original semantic inpainting framework incorporates only information from direct measurements, while in geostatistics indirect measurements are often plentiful. To overcome this limitation, here we propose a physics-informed…

TarGAN: Target-Aware Generative Adversarial Networks for Multi-modality Medical Image Translation

Paired multi-modality medical images, can provide complementary information to help physicians make more reasonable decisions than single modality medical images. But they are difficult to generate due to multiple factors in practice (e.g., time, cost, radiation dose). To address these problems, multi-modality medical image translation has aroused increasing research interest recently. However, the existing works mainly focus on translation effect of a whole image instead of a critical target area or Region of Interest (ROI), e.g., organ and so on. This leads to poor-quality translation of the…

PromptFlare：基于扩散修复中交叉注意力诱饵的提示泛化防御

PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting

PromptFlare 通过向提示中语义无信息的共享 token 注入交叉注意力诱饵噪声来抑制扩散修复模型的采样过程，从而在 EditBench 上以更低计算和显存开销实现最先进的图像防篡改保护。

Image Inpainting with Learnable Feature Imputation

A regular convolution layer applying a filter in the same way over known and unknown areas causes visual artifacts in the inpainted image. Several studies address this issue with feature re-normalization on the output of the convolution. However, these models use a significant amount of learnable parameters for feature re-normalization, or assume a binary representation of the certainty of an output. We propose (layer-wise) feature imputation of the missing input values to a convolution. In contrast to learned feature re-normalization, our method is efficient and introduces a minimal number of…

双模态医学图像合成：基于半监督序列生成对抗网络

Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks

提出一种基于半监督序列生成对抗网络的双模态医学图像合成方法，通过复杂度排序和联合/边缘分布学习，实现优于现有方法的合成效果。

DiffGANPaint：使用去噪扩散GAN的快速图像修复

DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs

本文提出基于去噪扩散概率模型（DDPM）的DiffGANPaint图像修复方法，利用GAN生成器建模反向扩散过程以降低采样成本，实现快速填充缺失像素，实验表明其性能优于或与多数当代工作持平。

改进MRI修复精度的后处理方法

Post-Processing Methods for Improving Accuracy in MRI Inpainting

本文提出结合模型集成与高效后处理策略的MRI修复方法，通过轻量U-Net增强阶段提升解剖合理性与视觉保真度，实现更准确且鲁棒的修复结果。