We present a scene representation, which we call a tactile-augmented radiance field (TaRF), that brings vision and touch into a shared 3D space. This representation can be used to estimate the visual and tactile signals for a given 3D position within a scene. We capture a scene's TaRF from a collection of photos and sparsely sampled touch probes. Our approach makes use of two insights: (i) common vision-based touch sensors are built on ordinary cameras and thus can be registered to images using methods from multi-view geometry, and (ii) visually and structurally similar regions of a scene share the same tactile features. We use these insights to register touch signals to a captured visual scene, and to train a conditional diffusion model that, provided with an RGB-D image rendered from a neural radiance field, generates its corresponding tactile signal. To evaluate our approach, we collect a dataset of TaRFs. This dataset contains more touch samples than previous real-world datasets, and it provides spatially aligned visual signals for each captured touch signal. We demonstrate the accuracy of our cross-modal generative model and the utility of the captured visual-tactile data on several downstream tasks. Project page: https://dou-yiming.github.io/TaRF
UC Los Angeles（加州大学洛杉矶分校）；University of Maryland（马里兰大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Columbia University（哥伦比亚大学）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Existing approaches to system identification (estimating the physical parameters of an object) from videos assume known object geometries. This precludes their applicability in a vast majority of scenes where object geometries are complex or unknown. In this work, we aim to identify parameters characterizing a physical system from a set of multi-view videos without any assumption on object geometry or topology. To this end, we propose "Physics Augmented Continuum Neural Radiance Fields" (PAC-NeRF), to estimate both the unknown geometry and physical parameters of highly dynamic objects from mul…
Apple（苹果公司）；The University of British Columbia（不列颠哥伦比亚大学）
Photorealistic rendering and reposing of humans is important for enabling augmented reality experiences. We propose a novel framework to reconstruct the human and the scene that can be rendered with novel human poses and views from just a single in-the-wild video. Given a video captured by a moving camera, we train two NeRF models: a human NeRF model and a scene NeRF model. To train these models, we rely on existing methods to estimate the rough geometry of the human and the scene. Those rough geometry estimates allow us to create a warping field from the observation space to the canonical pos…
This work investigates the use of Neural implicit representations, specifically Neural Radiance Fields (NeRF), for geometrical queries and motion planning. We show that by adding the capacity to infer occupancy in a radius to a pre-trained NeRF, we are effectively learning an approximation to a Euclidean Signed Distance Field (ESDF). Using backward differentiation of the augmented network, we obtain an obstacle gradient that is integrated into an obstacle avoidance policy based on the Riemannian Motion Policies (RMP) framework. Thus, our findings allow for very fast sampling-free obstacle avoi…
Neural Radiance Field (NeRF) regresses a neural parameterized scene by differentially rendering multi-view images with ground-truth supervision. However, when interpolating novel views, NeRF often yields inconsistent and visually non-smooth geometric results, which we consider as a generalization gap between seen and unseen views. Recent advances in convolutional neural networks have demonstrated the promise of advanced robust data augmentations, either random or learned, in enhancing both in-distribution and out-of-distribution generalization. Inspired by that, we propose Augmented NeRF (Aug-…
ETH Zurich（苏黎世联邦理工学院）；Google（谷歌）；TU Munich（慕尼黑工业大学）
Neural Radiance Fields (NeRF) coupled with GANs represent a promising direction in the area of 3D reconstruction from a single view, owing to their ability to efficiently model arbitrary topologies. Recent work in this area, however, has mostly focused on synthetic datasets where exact ground-truth poses are known, and has overlooked pose estimation, which is important for certain downstream applications such as augmented reality (AR) and robotics. We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available. Our approach…
CAD-Free Learning of Spacecraft Pose Estimators via NeRF-Based Augmentations
Department of Electrical Engineering (ELEN), ICTEAM, UCLouvain（电子工程系（ELEN），ICTEAM，鲁汶大学）；Department of Electrical Engineering (ESAT), KU Leuven（电子工程系（ESAT），鲁汶大学）；Department of Mechanical Engineering (MECH), KU Leuven（机械工程系（MECH），鲁汶大学）；Aerospacelab（航天实验室）
Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）
Driving scenes are extremely diverse and complicated that it is impossible to collect all cases with human effort alone. While data augmentation is an effective technique to enrich the training data, existing methods for camera data in autonomous driving applications are confined to the 2D image plane, which may not optimally increase data diversity in 3D real-world scenarios. To this end, we propose a 3D data augmentation approach termed Drive-3DAug, aiming at augmenting the driving scenes on camera in the 3D space. We first utilize Neural Radiance Field (NeRF) to reconstruct the 3D models of…
3D Gaussian Splatting against Moving Objects for High-Fidelity Street Scene Reconstruction
Ontario Knowledge Intelligence Centre, Canada（加拿大安大略知识智能中心）；Canadian Institute of Interscience, Canada（加拿大跨学科科学研究所）；DeepCox Intelligence, Canada（加拿大DeepCox智能科技公司）
School of Computer Science, Fudan University（复旦大学计算机科学学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Evaluating Human Perception of Novel View Synthesis: Subjective Quality Assessment of Gaussian Splatting and NeRF in Dynamic Scenes
College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）；School of Electronics and Communication Engineering, Guangzhou University（广州大学电子与通信工程学院）；Univ Rennes（雷恩大学）；INSA Rennes（雷恩国立应用科学学院）；CNRS（法国国家科学研究中心）；IRT b < > absent <> < > com；IETR - UMR 6164
