Image inpainting aims at restoring missing regions of corrupted images, which has many applications such as image restoration and object removal. However, current GAN-based generative inpainting models do not explicitly exploit the structural or textural consistency between restored contents and their surrounding contexts.To address this limitation, we propose to enforce the alignment (or closeness) between the local data submanifolds (or subspaces) around restored images and those around the original (uncorrupted) images during the learning process of GAN-based inpainting models. We exploit Local Intrinsic Dimensionality (LID) to measure, in deep feature space, the alignment between data submanifolds learned by a GAN model and those of the original data, from a perspective of both images (denoted as iLID) and local patches (denoted as pLID) of images. We then apply iLID and pLID as regularizations for GAN-based inpainting models to encourage two levels of submanifold alignment: 1) an image-level alignment for improving structural consistency, and 2) a patch-level alignment for improving textural details. Experimental results on four benchmark datasets show that our proposed model can generate more accurate results than state-of-the-art models.
Most existing image inpainting algorithms are based on a single view, struggling with large holes or the holes containing complicated scenes. Some reference-guided algorithms fill the hole by referring to another viewpoint image and use 2D image alignment. Due to the camera imaging process, simple 2D transformation is difficult to achieve a satisfactory result. In this paper, we propose 3DFill, a simple and efficient method for reference-guided image inpainting. Given a target image with arbitrary hole regions and a reference image from another viewpoint, the 3DFill first aligns the two images…
MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting
MT Lab, Meitu Inc（美图实验室，美图公司）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Technology, BNRist, IDG/McGovern Institute for Brain Research, Tsinghua University（计算机科学与技术系，BNRist，IDG/McGovern脑科学研究院，清华大学）
We present a novel deep learning based algorithm for video inpainting. Video inpainting is a process of completing corrupted or missing regions in videos. Video inpainting has additional challenges compared to image inpainting due to the extra temporal information as well as the need for maintaining the temporal coherency. We propose a novel DNN-based framework called the Copy-and-Paste Networks for video inpainting that takes advantage of additional information in other frames of the video. The network is trained to copy corresponding contents in reference frames and paste them to fill the ho…
Australian AI Institute（澳大利亚人工智能研究所）；University of Technology Sydney（悉尼科技大学）
Text-guided image inpainting (TGII) aims to restore missing regions based on a given text in a damaged image. Existing methods are based on a strong vision encoder and a cross-modal fusion model to integrate cross-modal features. However, these methods allocate most of the computation to visual encoding, while light computation on modeling modality interactions. Moreover, they take cross-modal fusion for depth features, which ignores a fine-grained alignment between text and image. Recently, vision-language pre-trained models (VLPM), encapsulating rich cross-modal alignment knowledge, have adv…
Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator
Data Science and AI Laboratory, ECE, Seoul National University（数据科学与人工智能实验室、电子与计算机工程系、首尔国立大学）；AIIS, ASRI, INMC, ISRC, and Interdisciplinary Program in AI, Seoul National University（人工智能研究所、人工智能研究室、智能网络与计算中心、信息科学与工程系以及人工智能交叉学科项目、首尔国立大学）
Wuhan University（武汉大学）；Nanyang Technological University（南洋理工大学）；National Institute of Informatics（国立信息学研究所）
Existing learning-based image inpainting methods are still in challenge when facing complex semantic environments and diverse hole patterns. The prior information learned from the large scale training data is still insufficient for these situations. Reference images captured covering the same scenes share similar texture and structure priors with the corrupted images, which offers new prospects for the image inpainting tasks. Inspired by this, we first build a benchmark dataset containing 10K pairs of input and reference images for reference-guided inpainting. Then we adopt an encoder-decoder …
CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility
The Chinese University of Hong Kong（香港中文大学）；International Digital Economy Academy (IDEA)（国际数字经济学院）；The University of Hong Kong（香港大学）；Tsinghua University（清华大学）
SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpainting
Department of Electrical and Computer Engineering, Aristotle University of Thessaloniki（阿尔伯塔大学电气与计算机工程系）；Information Technologies Institute, CERTH（信息科技研究所）
Facebook Artificial Intelligence（脸书人工智能）；University of Southern California（南加州大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Deformable registration of magnetic resonance images between patients with brain tumors and healthy subjects has been an important tool to specify tumor geometry through location alignment and facilitate pathological analysis. Since tumor region does not match with any ordinary brain tissue, it has been difficult to deformably register a patients brain to a normal one. Many patient images are associated with irregularly distributed lesions, resulting in further distortion of normal tissue structures and complicating registration's similarity measure. In this work, we follow a multi-step contex…
Many video editing tasks such as rotoscoping or object removal require the propagation of context across frames. While transformers and other attention-based approaches that aggregate features globally have demonstrated great success at propagating object masks from keyframes to the whole video, they struggle to propagate high-frequency details such as textures faithfully. We hypothesize that this is due to an inherent bias of global attention towards low-frequency features. To overcome this limitation, we present a two-stream approach, where high-frequency features interact locally and low-fr…
MAGIC: Few-Shot Mask-Guided Anomaly Inpainting with Prompt Perturbation, Spatially Adaptive Guidance, and Context Awareness
Department of AI Semiconductor Engineering, Hanyang University, Seoul, Korea（汉阳大学人工智能半导体工程系）；Department of Electronic Engineering, Hanyang University, Seoul, Korea（汉阳大学电子工程系）
Image inpainting techniques have shown promising improvement with the assistance of generative adversarial networks (GANs) recently. However, most of them often suffered from completed results with unreasonable structure or blurriness. To mitigate this problem, in this paper, we present a one-stage model that utilizes dense combinations of dilated convolutions to obtain larger and more effective receptive fields. Benefited from the property of this network, we can more easily recover large regions in an incomplete image. To better train this efficient generator, except for frequently-used VGG …
ByteDance Inc.（字节跳动有限公司）；University of California, Davis（加利福尼亚大学戴维斯分校）
Video inpainting aims to fill spatio-temporal "corrupted" regions with plausible content. To achieve this goal, it is necessary to find correspondences from neighbouring frames to faithfully hallucinate the unknown content. Current methods achieve this goal through attention, flow-based warping, or 3D temporal convolution. However, flow-based warping can create artifacts when optical flow is not accurate, while temporal convolution may suffer from spatial misalignment. We propose 'Progressive Temporal Feature Alignment Network', which progressively enriches features extracted from the current …
Hairstyle transfer is the task of modifying a source hairstyle to a target one. Although recent hairstyle transfer models can reflect the delicate features of hairstyles, they still have two major limitations. First, the existing methods fail to transfer hairstyles when a source and a target image have different poses (e.g., viewing direction or face size), which is prevalent in the real world. Also, the previous models generate unrealistic images when there is a non-trivial amount of regions in the source image occluded by its original hair. When modifying long hair to short hair, shoulders o…
Key Laboratory of Ministry of Education for Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（教育部图像处理与智能控制重点实验室，人工智能与自动化学院，华中科技大学）；Alibaba Group（阿里巴巴集团）；Nanyang Technological University（南洋理工大学）；Nanjing University（南京大学）；Zhejiang University（浙江大学）
本文提出 ReplaceAnyone 框架，将局部人物替换与插入任务形式化为姿态引导的图像条件视频修复，通过多样化掩码、增强视觉引导、混合修复编码器和两阶段优化，在保留背景与运动的同时实现无缝人物替换，并兼容 3D-UNet 和 DiT 视频模型。
Kuaishou Technology（快手科技）；SIGS, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng National Laboratory（鹏城国家实验室）
This paper proposes a novel video inpainting method. We make three main contributions: First, we extended previous Transformers with patch alignment by introducing Deformed Patch-based Homography (DePtH), which improves patch-level feature alignments without additional supervision and benefits challenging scenes with various deformation. Second, we introduce Mask Pruning-based Patch Attention (MPPA) to improve patch-wised feature matching by pruning out less essential features and using saliency map. MPPA enhances matching accuracy between warped tokens with invalid pixels. Third, we introduce…
Nara Institute of Science and Technology（奈良先端科学技术大学院大学）
In this work we propose a novel approach to remove undesired objects from RGB-D sequences captured with freely moving cameras, which enables static 3D reconstruction. Our method jointly uses existing information from multiple frames as well as generates new one via inpainting techniques. We use balanced rules to select source frames; local homography based image warping method for alignment and Markov random field (MRF) based approach for combining existing information. For the left holes, we employ exemplar based multi-view inpainting method to deal with the color image and coherently use it …
In this paper, we focus on person image generation, namely, generating person image under various conditions, e.g., corrupted texture or different pose. To address texture occlusion and large pose misalignment in this task, previous works just use the corresponding region's style to infer the occluded area and rely on point-wise alignment to reorganize the context texture information, lacking the ability to globally correlate the region-wise style codes and preserve the local structure of the source. To tackle these problems, we present a GLocal framework to improve the occlusion-aware texture…
We present Poly-GAN, a novel conditional GAN architecture that is motivated by Fashion Synthesis, an application where garments are automatically placed on images of human models at an arbitrary pose. Poly-GAN allows conditioning on multiple inputs and is suitable for many tasks, including image alignment, image stitching, and inpainting. Existing methods have a similar pipeline where three different networks are used to first align garments with the human pose, then perform stitching of the aligned garment and finally refine the results. Poly-GAN is the first instance where a common architect…
Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignment
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China（电子与电气工程系，南方科技大学，深圳，中国）；Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China（电气与电子工程系，香港大学，香港特别行政区，中国）；Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing, China（放射科，中大医院，医学院，东南大学，南京，中国）；Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, China（嘉兴研究所，南方科技大学，嘉兴，中国）
PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data
National University of Singapore（新加坡国立大学）；Shanghai Jiao Tong University（上海交通大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tiamat（提亚马特）
