Image completion has made tremendous progress with convolutional neural networks (CNNs), because of their powerful texture modeling capacity. However, due to some inherent properties (e.g., local inductive prior, spatial-invariant kernels), CNNs do not perform well in understanding global structures or naturally support pluralistic completion. Recently, transformers demonstrate their power in modeling the long-term relationship and generating diverse results, but their computation complexity is quadratic to input length, thus hampering the application in processing high-resolution images. This paper brings the best of both worlds to pluralistic image completion: appearance prior reconstruction with transformer and texture replenishment with CNN. The former transformer recovers pluralistic coherent structures together with some coarse textures, while the latter CNN enhances the local texture details of coarse priors guided by the high-resolution masked images. The proposed method vastly outperforms state-of-the-art methods in terms of three aspects: 1) large performance boost on image fidelity even compared to deterministic completion methods; 2) better diversity and higher fidelity for pluralistic completion; 3) exceptional generalization ability on large masks and generic dataset, like ImageNet.
The University of Sydney（悉尼大学）；Nanjing University（南京大学）；The University of Edinburgh（爱丁堡大学）；Nanyang Technological University（南洋理工大学）；JD Explore Academy（京东探索研究院）；Hong Kong Baptist University（香港浸会大学）
Pluralistic image completion focuses on generating both visually realistic and diverse results for image completion. Prior methods enjoy the empirical successes of this task. However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects. First, the constraints for visual reality can be weakly correlated to the objective of image completion or even redundant. Second, the constraints for diversity are designed to be task-agnostic, which causes the constraints to not work well. In this paper, to address the issues, we …
Most image completion methods produce only one result for each masked input, although there may be many reasonable possibilities. In this paper, we present an approach for \textbf{pluralistic image completion} -- the task of generating multiple and diverse plausible solutions for image completion. A major challenge faced by learning-based approaches is that usually only one ground truth training instance per label. As such, sampling from conditional VAEs still leads to minimal diversity. To overcome this, we propose a novel and probabilistically principled framework with two parallel paths. On…
PICNet pioneered the generation of multiple and diverse results for image completion task, but it required a careful balance between $\mathcal{KL}$ loss (diversity) and reconstruction loss (quality), resulting in a limited diversity and quality . Separately, iGPT-based architecture has been employed to infer distributions in a discrete space derived from a pixel-level pre-clustered palette, which however cannot generate high-quality results directly. In this work, we present a novel framework for pluralistic image completion that can achieve both high quality and diversity at much faster infer…
Transformer based Pluralistic Image Completion with Reduced Information Loss
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Microsoft Cloud AI（微软云人工智能部门）；Wormpex AI Research LLC（Wormpex人工智能研究有限责任公司）
University of Science and Technology of China（中国科学技术大学）；Microsoft（微软）
Transformers have achieved great success in pluralistic image inpainting recently. However, we find existing transformer based solutions regard each pixel as a token, thus suffer from information loss issue from two aspects: 1) They downsample the input image into much lower resolutions for efficiency consideration, incurring information loss and extra misalignment for the boundaries of masked regions. 2) They quantize $256^3$ RGB pixels to a small number (such as 512) of quantized pixels. The indices of quantized pixels are used as tokens for the inputs and prediction targets of transformer. …
Bridging global context interactions correctly is important for high-fidelity image completion with large masks. Previous methods attempting this via deep or large receptive field (RF) convolutions cannot escape from the dominance of nearby interactions, which may be inferior. In this paper, we propose to treat image completion as a directionless sequence-to-sequence prediction task, and deploy a transformer to directly capture long-range dependence in the encoder. Crucially, we employ a restrictive CNN with small and non-overlapping RF for weighted token representation, which allows the trans…
FaceCom: Towards High-fidelity 3D Facial Shape Completion via Optimization and Inpainting Guidance
Beihang University（北京航空航天大学）；Southwest University（西南大学）；Peking University School and Hospital of Stomatology（北京大学口腔医学院）；State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统国家重点实验室）
South China University of Technology（华南理工大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Sydney（悉尼大学）
High-fidelity face completion is a challenging task due to the rich and subtle facial textures involved. What makes it more complicated is the correlations between different facial components, for example, the symmetry in texture and structure between both eyes. While recent works adopted the attention mechanism to learn the contextual relations among elements of the face, they have largely overlooked the disastrous impacts of inaccurate attention scores; in addition, they fail to pay sufficient attention to key facial components, the completion results of which largely determine the authentic…
Semantic shape completion is a challenging problem in 3D computer vision where the task is to generate a complete 3D shape using a partial 3D shape as input. We propose a learning-based approach to complete incomplete 3D shapes through generative modeling and latent manifold optimization. Our algorithm works directly on point clouds. We use an autoencoder and a GAN to learn a distribution of embeddings for point clouds of object classes. An input point cloud with missing regions is first encoded to a feature vector. The representations learnt by the GAN are then used to find the best latent ve…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent America（腾讯美国）
Completing an unordered partial point cloud is a challenging task. Existing approaches that rely on decoding a latent feature to recover the complete shape, often lead to the completed point cloud being over-smoothing, losing details, and noisy. Instead of decoding a whole shape, we propose to decode and refine a low-resolution (low-res) point cloud first, and then performs a patch-wise noise-aware upsampling rather than interpolating the whole sparse point cloud at once, which tends to lose details. Regarding the possibility of lacking details of the initially decoded low-res point cloud, we …
3DMambaComplete: Exploring Structured State Space Model for Point Cloud Completion
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University（复旦大学计算机学院数据科学实验室）；Department of Information Engineering, The Chinese University of Hong Kong（香港中文大学信息工程系）
IIIS, Tsinghua University（清华大学交叉信息研究院）；Microsoft Research（微软研究院）；Vacaville Christian Schools（瓦卡维尔基督学校）；The High School Affiliated to Renmin University of China（中国人民大学附属中学）
Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Incepti…
SynergyAmodal: Deocclude Anything with Text Control
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, China（多媒体可信感知与高效计算重点实验室、教育部、厦门大学）
In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel weakly-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significa…
Delft University of Technology（代尔夫特理工大学）；The Chinese University of Hong Kong（香港中文大学）；South China University of Technology（华南理工大学）
Using the raw data from consumer-level RGB-D cameras as input, we propose a deep-learning based approach to efficiently generate RGB-D images with completed information in high resolution. To process the input images in low resolution with missing regions, new operators for adaptive convolution are introduced in our deep-learning network that consists of three cascaded modules -- the completion module, the refinement module and the super-resolution module. The completion module is based on an architecture of encoder-decoder, where the features of input raw RGB-D will be automatically extracted…
InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；The Hong Kong University of Science and Technology（香港科技大学）；Ant Group（蚂蚁集团）
3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image
College of Computer Science, Nankai University（南开大学计算机科学学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；Horizon Robotics（地平线机器人）；D-Robotics
VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（可信自主系统研究院，南方科技大学）；Department of Computer Science and Engineering, Southern University of Science and Technology（计算机科学与工程系，南方科技大学）；School of Computer Science, University of Nottingham Ningbo China（宁波大学计算机学院）；Department of Electronic and Information Engineering, Changchun University（电子与信息工程学院，长春大学）
Trustworthy Longitudinal Brain MRI Completion: A Deformation-Based Approach with KAN-Enhanced Diffusion Model
School of Biomedical Engineering and Imaging Science, King's College London, London, UK（生物医学工程与成像科学学院，伦敦国王学院）；School of Biomedical Engineering, ShanghaiTech University, Shanghai, China（生物医学工程学院，上海科技大学）；School of Computer Science and Digital Technologies, Aston University, Birmingham, UK（计算机科学与数字技术学院，阿斯顿大学）；Kunming Medical University, Kunming, China（昆明医学院）
VGG-Tex: A Vivid Geometry-Guided Facial Texture Estimation Model for High Fidelity Monocular 3D Face Reconstruction
Renmin University of China（中国人民大学）；Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Beijing Academy of Blockchain and Edge Computing（北京区块链与边缘计算研究院）
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing Institute of Software Technology（南京软件技术研究院）；University of North Texas（北得克萨斯大学）
Image inpainting seeks a semantically consistent way to recover the corrupted image in the light of its unmasked content. Previous approaches usually reuse the well-trained GAN as effective prior to generate realistic patches for missing holes with GAN inversion. Nevertheless, the ignorance of a hard constraint in these algorithms may yield the gap between GAN inversion and image inpainting. Addressing this problem, in this paper, we devise a novel GAN inversion model for image inpainting, dubbed InvertFill, mainly consisting of an encoder with a pre-modulation module and a GAN generator with …
LoCI-DiffCom: Longitudinal Consistency-Informed Diffusion Model for 3D Infant Brain Image Completion
School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trail Center（上海临床研究中心）
StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion
Nanjing University of Posts and Telecommunications（南京邮电大学）；Peng Cheng Laboratory（鹏城实验室）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of the Chinese Academy of Sciences（中国科学院大学人工智能学院）
FlowObject: Flow Steering for Bridging Generative Priors and Reconstruction Fidelity
Graz University of Technology Austria（奥地利格拉茨理工大学）；Tampere University Finland（芬兰塔尔库大学）；Technical University of Munich Germany（德国慕尼黑技术大学）；Stanford University The United States of America（美国斯坦福大学）；Xi’an Jiaotong University China（中国西安交通大学）；École des Ponts ParisTech France（法国巴黎综合理工学院）
Gated imaging is an emerging sensor technology for self-driving cars that provides high-contrast images even under adverse weather influence. It has been shown that this technology can even generate high-fidelity dense depth maps with accuracy comparable to scanning LiDAR systems. In this work, we extend the recent Gated2Depth framework with aleatoric uncertainty providing an additional confidence measure for the depth estimates. This confidence can help to filter out uncertain estimations in regions without any illumination. Moreover, we show that training on dense depth maps generated by LiD…
NeRF provides unparalleled fidelity of novel view synthesis: rendering a 3D scene from an arbitrary viewpoint. NeRF requires training on a large number of views that fully cover a scene, which limits its applicability. While these issues can be addressed by learning a prior over scenes in various forms, previous approaches have been either applied to overly simple scenes or struggling to render unobserved parts. We introduce Laser-NV: a generative model which achieves high modelling capacity, and which is based on a set-valued latent representation modelled by normalizing flows. Similarly to p…
ReWorld: Multi-Dimensional Reward Modeling for Embodied World Models
Eastern Institute of Technology（东部技术研究所）；Georgia Institute of Technology（佐治亚理工学院）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）；Peking University（北京大学）
Image inpainting is an underdetermined inverse problem, which naturally allows diverse contents to fill up the missing or corrupted regions realistically. Prevalent approaches using convolutional neural networks (CNNs) can synthesize visually pleasant contents, but CNNs suffer from limited perception fields for capturing global features. With image-level attention, transformers enable to model long-range dependencies and generate diverse contents with autoregressive modeling of pixel-sequence distributions. However, the unidirectional attention in autoregressive transformers is suboptimal as c…
