We present a classification based approach for the next best view selection and show how we can plausibly obtain a supervisory signal for this task. The proposed approach is end-to-end trainable and aims to get the best possible 3D reconstruction quality with a pair of passively acquired 2D views. The proposed model consists of two stages: a classifier and a reconstructor network trained jointly via the indirect 3D supervision from ground truth voxels. While testing, the proposed method assumes no prior knowledge of the underlying 3D shape for selecting the next best view. We demonstrate the proposed method's effectiveness via detailed experiments on synthetic and real images and show how it provides improved reconstruction quality than the existing state of the art 3D reconstruction and the next best view prediction techniques.
Novel View Synthesis (NVS) is concerned with synthesizing views under camera viewpoint transformations from one or multiple input images. NVS requires explicit reasoning about 3D object structure and unseen parts of the scene to synthesize convincing results. As a result, current approaches typically rely on supervised training with either ground truth 3D models or multiple target images. We propose Continuous Object Representation Networks (CORN), a conditional architecture that encodes an input image's geometry and appearance that map to a 3D consistent scene representation. We can train COR…
NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References
School of Computer Science, the University of Sydney（悉尼大学计算机科学学院）；School of Software, Shandong University（山东大学软件学院）；School of Computer and Artificial Intelligence, Beijing Technology and Business University（北京科技与商业大学计算机与人工智能学院）
Physically Inspired Gaussian Splatting for HDR Novel View Synthesis
Department of Electrical and Computer Engineering, Northeastern University（东北大学电气与计算机工程系）；Khoury College of Computer Science, Northeastern University（东北大学Khoury计算机科学学院）
Novel view synthesis (NVS) and video prediction (VP) are typically considered disjoint tasks in computer vision. However, they can both be seen as ways to observe the spatial-temporal world: NVS aims to synthesize a scene from a new point of view, while VP aims to see a scene from a new point of time. These two tasks provide complementary signals to obtain a scene representation, as viewpoint changes from spatial observations inform depth, and temporal observations inform the motion of cameras and individual objects. Inspired by these observations, we propose to study the problem of Video Extr…
Harvard University（哈佛大学）；Facebook AI（脸书人工智能）；Carnegie Mellon University（卡内基梅隆大学）；The Hong Kong Polytechnic University（香港理工大学）；Université de Montréal（蒙特利尔大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Fanhan Tech. Inc.（泛函科技有限公司）
This paper targets on learning-based novel view synthesis from a single or limited 2D images without the pose supervision. In the viewer-centered coordinates, we construct an end-to-end trainable conditional variational framework to disentangle the unsupervisely learned relative-pose/rotation and implicit global 3D representation (shape, texture and the origin of viewer-centered coordinates, etc.). The global appearance of the 3D object is given by several appearance-describing images taken from any number of viewpoints. Our spatial correlation module extracts a global 3D representation from t…
D$^2$GS: Depth-and-Density Guided Gaussian Splatting for Stable and Accurate Sparse-View Reconstruction
Insta360 Research（Insta360研究机构）；Tsinghua University（清华大学）；University of California, San Diego（加州大学圣地亚哥分校）；Nanyang Technological University（南洋理工大学）；Wuhan University（武汉大学）
Resolving Representation Ambiguity in Feedforward Novel View Synthesis Transformer via Semantic-Spatial Decoupling
Institute of Trustworthy Embodied Artificial Intelligence (TEAI)（可信具身人工智能研究所）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）；Sch. of Artificial Intelligence & Sch. of Computer Science, Shanghai Jiao Tong University（上海交通大学人工智能学院与计算机科学学院）；University of Science and Technology of China（中国科学技术大学）
Australian National University and ACRV（澳大利亚国立大学与ACRV）；University of Technology Sydney（悉尼科技大学）
We address the problem of novel view synthesis (NVS) from a few sparse source view images. Conventional image-based rendering methods estimate scene geometry and synthesize novel views in two separate steps. However, erroneous geometry estimation will decrease NVS performance as view synthesis highly depends on the quality of estimated scene geometry. In this paper, we propose an end-to-end NVS framework to eliminate the error propagation issue. To be specific, we construct a volume under the target view and design a source-view visibility estimation (SVE) module to determine the visibility of…
Department of Computer Science and Engineering, IIT Kanpur（印度理工学院坎普尔分校计算机科学与工程系）
Multiple categories of objects are present in most images. Treating this as a multi-class classification is not justified. We treat this as a multi-label classification problem. In this paper, we further aim to minimize the supervision required for providing supervision in multi-label classification. Specifically, we investigate an effective class of approaches that associate a weak localization with each category either in terms of the bounding box or segmentation mask. Doing so improves the accuracy of multi-label categorization. The approach we adopt is one of active learning, i.e., increme…
TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction
VCIP, CS, Nankai University（南开大学计算机科学与技术学院）；NKIARI, Shenzhen Futian（深圳未来科技研究院）；USTC（University of Science and Technology of China）；CUHK MMLab（香港中文大学MMLab）；VAST（中国科学院自动化研究所）；Shanghai AI Lab（上海人工智能实验室）
