In this work, we present a novel framework built to simplify 3D asset generation for amateur users. To enable interactive generation, our method supports a variety of input modalities that can be easily provided by a human, including images, text, partially observed shapes and combinations of these, further allowing to adjust the strength of each input. At the core of our approach is an encoder-decoder, compressing 3D shapes into a compact latent representation, upon which a diffusion model is learned. To enable a variety of multi-modal inputs, we employ task-specific encoders with dropout followed by a cross-attention mechanism. Due to its flexibility, our model naturally supports a variety of tasks, outperforming prior works on shape completion, image-based 3D reconstruction, and text-to-3D. Most interestingly, our model can combine all these tasks into one swiss-army-knife tool, enabling the user to perform shape generation using incomplete shapes, images, and textual descriptions at the same time, providing the relative weights for each input and facilitating interactivity. Despite our approach being shape-only, we further show an efficient method to texture the generated shape using large-scale text-to-image models.
Several deep learning methods have been proposed for completing partial data from shape acquisition setups, i.e., filling the regions that were missing in the shape. These methods, however, only complete the partial shape with a single output, ignoring the ambiguity when reasoning the missing geometry. Hence, we pose a multi-modal shape completion problem, in which we seek to complete the partial shape with multiple outputs by learning a one-to-many mapping. We develop the first multimodal shape completion method that completes the partial shape via conditional generative modeling, without req…
UC Santa Barbara（加州大学圣巴巴拉分校）；Meta；NEC Laboratories America（NEC美国实验室）
Generating a video given the first several static frames is challenging as it anticipates reasonable future frames with temporal coherence. Besides video prediction, the ability to rewind from the last frame or infilling between the head and tail is also crucial, but they have rarely been explored for video completion. Since there could be different outcomes from the hints of just a few frames, a system that can follow natural language to perform video completion may significantly improve controllability. Inspired by this, we introduce a novel task, text-guided video completion (TVC), which re…
MIT Lincoln Laboratory（麻省理工学院林肯实验室）；MIT（麻省理工学院）；DAF MIT AI Accelerator（美国空军部麻省理工学院人工智能加速器）
The Multimodal Learning for Earth and Environment Challenge (MultiEarth 2022) will be the first competition aimed at the monitoring and analysis of deforestation in the Amazon rainforest at any time and in any weather conditions. The goal of the Challenge is to provide a common benchmark for multimodal information processing and to bring together the earth and environmental science communities as well as multimodal representation learning communities to compare the relative merits of the various multimodal learning methods to deforestation estimation under well-defined and strictly comparable …
UC Santa Barbara（加州大学圣巴巴拉分校）；UC Santa Cruz（加州大学圣克鲁兹分校）；UC San Diego（加州大学圣地亚哥分校）；Google（谷歌）
One of the most challenging topics in Natural Language Processing (NLP) is visually-grounded language understanding and reasoning. Outdoor vision-and-language navigation (VLN) is such a task where an agent follows natural language instructions and navigates a real-life urban environment. Due to the lack of human-annotated instructions that illustrate intricate urban scenes, outdoor VLN remains a challenging task to solve. This paper introduces a Multimodal Text Style Transfer (MTST) learning approach and leverages external multimodal resources to mitigate data scarcity in outdoor navigation ta…
Earth observation satellites have been continuously monitoring the earth environment for years at different locations and spectral bands with different modalities. Due to complex satellite sensing conditions (e.g., weather, cloud, atmosphere, orbit), some observations for certain modalities, bands, locations, and times may not be available. The MultiEarth Matrix Completion Challenge in CVPR 2022 [1] provides the multimodal satellite data for addressing such data sparsity challenges with the Amazon Rainforest as the region of interest. This work proposes an adaptive real-time multimodal regress…
Simon Fraser University（西蒙弗雷泽大学）；Flipkart（Flipkart公司）
Shape completion is the problem of completing partial input shapes such as partial scans. This problem finds important applications in computer vision and robotics due to issues such as occlusion or sparsity in real-world data. However, most of the existing research related to shape completion has been focused on completing shapes by learning a one-to-one mapping which limits the diversity and creativity of the produced results. We propose a novel multimodal shape completion technique that is effectively able to learn a one-to-many mapping and generates diverse complete shapes. Our approach is…
Peking University（北京大学）；Chongqing University（重庆大学）；Tsinghua University（清华大学）；Hefei University of Technology（合肥工业大学）
With the wide application of sparse ToF sensors in mobile devices, RGB image-guided sparse depth completion has attracted extensive attention recently, but still faces some problems. First, the fusion of multimodal information requires more network modules to process different modalities. But the application scenarios of sparse ToF measurements usually demand lightweight structure and low computational cost. Second, fusing sparse and noisy depth data with dense pixel-wise RGB data may introduce artifacts. In this paper, a light but efficient depth completion network is proposed, which consists…
Depth prediction is a critical problem in robotics applications especially autonomous driving. Generally, depth prediction based on binocular stereo matching and fusion of monocular image and laser point cloud are two mainstream methods. However, the former usually suffers from overfitting while building cost volume, and the latter has a limited generalization due to the lack of geometric constraint. To solve these problems, we propose a novel multimodal neural network, namely UAMD-Net, for dense depth completion based on fusion of binocular stereo matching and the weak constrain from the spar…
MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所 multimodal 人工智能系统国家重点实验室）；Chemical Defense Institute, Academy of Military Sciences（军事科学院化学防御研究院）；Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所 复杂系统认知与决策智能重点实验室）
DuInNet: Dual-Modality Feature Interaction for Point Cloud Completion
College of Electronic Science and Technology, National University of Defense Technology (NUDT)（国防科技大学电子科学学院）；School of Surveying and Mapping, Information Engineering University（信息工程大学测绘学院）
Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model
Dalian Maritime University（大连海事大学）；Zhejiang Sci-Tech University（浙江理工大学）；Beijing University of Technology（北京工业大学）；Dalian University of Technology（大连理工大学）
Powerful priors allow us to perform inference with insufficient information. In this paper, we propose an autoregressive prior for 3D shapes to solve multimodal 3D tasks such as shape completion, reconstruction, and generation. We model the distribution over 3D shapes as a non-sequential autoregressive distribution over a discretized, low-dimensional, symbolic grid-like latent representation of 3D shapes. This enables us to represent distributions over 3D shapes conditioned on information from an arbitrary set of spatially anchored query locations and thus perform shape completion in such arbi…
We propose a novel method for 3D shape completion from a partial observation of a point cloud. Existing methods either operate on a global latent code, which limits the expressiveness of their model, or autoregressively estimate the local features, which is highly computationally extensive. Instead, our method estimates the entire local feature field by a single feedforward network by formulating this problem as a tensor completion problem on the feature volume of the object. Due to the redundancy of local feature volumes, this tensor completion problem can be further reduced to estimating the…
In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel weakly-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significa…
Recently, virtual/pseudo-point-based 3D object detection that seamlessly fuses RGB images and LiDAR data by depth completion has gained great attention. However, virtual points generated from an image are very dense, introducing a huge amount of redundant computation during detection. Meanwhile, noises brought by inaccurate depth completion significantly degrade detection precision. This paper proposes a fast yet effective backbone, termed VirConvNet, based on a new operator VirConv (Virtual Sparse Convolution), for virtual-point-based 3D object detection. VirConv consists of two key designs: …
A Strong View-Free Baseline Approach for Single-View Image Guided Point Cloud Completion
Department of Robotics Engineering, Worcester Polytechnic Institute（机器人工程系，沃斯特理工学院）；Dell Technologies（戴尔技术）；Graduate School of Information Sciences, Tohoku University（信息科学研究生院，东北大学）；Department of Computer Science, Worcester Polytechnic Institute（计算机科学系，沃斯特理工学院）；Harvard Kenneth C. Griffin Graduate School of Arts and Sciences（哈佛肯尼斯·C·格里芬艺术与科学研究生院）；Department of Biomedical Engineering, Worcester Polytechnic Institute（生物医学工程系，沃斯特理工学院）；Department of Electrical & Computer Engineering, Worcester Polytechnic Institute（电气与计算机工程系，沃斯特理工学院）
Knowledge Bridger: Towards Training-free Missing Modality Completion
Beijing Jiaotong University（北京交通大学）；Singapore Management University（新加坡国立大学）；Southeast University（东南大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Harbin Institute of Technology（哈尔滨工业大学）；Nanjing University of Science and Technology（南京理工大学）；South China University of Technology（华南理工大学）；Xiamen Institute of Technology（厦门理工学院）
Position-aware Guided Point Cloud Completion with CLIP Model
North China University of Technology（北方工业大学）；Peng Cheng Laboratory（鹏城实验室）；University of Copenhagen（哥本哈根大学）；University of Washington（华盛顿大学）；Skywork AI（天工智能）；Tsinghua University（清华大学）
Cross-modal alignment is essential for vision-language pre-training (VLP) models to learn the correct corresponding information across different modalities. For this purpose, inspired by the success of masked language modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-to-local alignment. However, most of them pay little attention to the global semantic fe…
