针对无纹理或弱纹理区域点跟踪难题，本文提出纹理强度指标，构建含1800个标注室内视频的 GIFT 合成基准，并系统评估现有方法。
摘要：
点跟踪正成为运动估计和视频编辑的强大解决方案。与经典特征匹配相比，点跟踪方法的关键优势在于能够在复杂相机运动轨迹下以及较长时间内鲁棒地跟踪点。然而，尽管方法学上已有一定改进，当前点跟踪方法仍难以跟踪视频帧中的任意位置，尤其是在无纹理或弱纹理区域。在本工作中，我们首先引入用于评估三维物体纹理强度的指标。利用这些指标，我们将 ShapeNet 中的三维模型划分为三个纹理强度等级，并创建了 GIFT，这是一个具有挑战性的合成基准，包含1800个带有丰富标注的室内视频序列。与现有数据集任意指定真值点不同，GIFT 将真值精确锚定在已分类的目标物体上，确保每个视频对应一个特定的纹理强度等级。此外，我们在 GIFT 上全面评估当前方法，以衡量它们在不同纹理强度等级上的性能，并分析纹理对点跟踪的影响。
英文摘要：
Point tracking is becoming a powerful solver for motion estimation and video editing. Compared to classical feature matching, point tracking methods have the key advantage of robustly tracking points under complex camera motion trajectories and over extended periods. However, despite certain improvements in methodologies, current point tracking methods still struggle to track any position in video frames, especially in areas that are texture-less or weakly textured. In this work, we first introduce metrics for evaluating the texture intensity of a 3D object. Using these metrics, we classify the 3D models in ShapeNet into three levels of texture intensity and create GIFT, a challenging synthetic benchmark comprising 1800 indoor video sequences with rich annotations. Unlike existing datasets that assign ground truth points arbitrarily, GIFT precisely anchors ground truth on classified target objects, ensuring that each video corresponds to a specific texture intensity level. Furthermore, we comprehensively evaluate current methods on GIFT to assess their performance across different texture intensity levels and analyze the impact of texture on point tracking.
The University of Hong Kong（香港大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Texas A&M University（得克萨斯农工大学）
Reconstructing 3D indoor scenes from 2D images is an important task in many computer vision and graphics applications. A main challenge in this task is that large texture-less areas in typical indoor scenes make existing methods struggle to produce satisfactory reconstruction results. We propose a new method, named NeuRIS, for high quality reconstruction of indoor scenes. The key idea of NeuRIS is to integrate estimated normal of indoor scenes as a prior in a neural rendering framework for reconstructing large texture-less shapes and, importantly, to do this in an adaptive manner to also enabl…
We present an end-to-end deep learning framework for indoor panoramic image inpainting. Although previous inpainting methods have shown impressive performance on natural perspective images, most fail to handle panoramic images, particularly indoor scenes, which usually contain complex structure and texture content. To achieve better inpainting quality, we propose to exploit both the global and local context of indoor panorama during the inpainting process. Specifically, we take the low-level layout edges estimated from the input panorama as a prior to guide the inpainting model for recovering …
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Alibaba-NTU Singapore Joint Research Institute, Nanyang Technological University（阿里巴巴-南洋理工大学新加坡联合研究院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Technical University of Munich（慕尼黑工业大学）；University of Michigan（密歇根大学）
We apply style transfer on mesh reconstructions of indoor scenes. This enables VR applications like experiencing 3D environments painted in the style of a favorite artist. Style transfer typically operates on 2D images, making stylization of a mesh challenging. When optimized over a variety of poses, stylization patterns become stretched out and inconsistent in size. On the other hand, model-based 3D style transfer methods exist that allow stylization from a sparse set of images, but they require a network at inference time. To this end, we optimize an explicit texture for the reconstructed me…
RDFC-GAN: RGB-Depth Fusion CycleGAN for Indoor Depth Completion
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, China（网络与交换技术国家重点实验室，北京邮电大学，中国）；Midea Group, China（美的集团，中国）；School of Computer Science, Beijing University of Posts and Telecommunications, China（计算机科学学院，北京邮电大学，中国）
We present a novel approach to reconstruct RGB-D indoor scene with plane primitives. Our approach takes as input a RGB-D sequence and a dense coarse mesh reconstructed by some 3D reconstruction method on the sequence, and generate a lightweight, low-polygonal mesh with clear face textures and sharp features without losing geometry details from the original scene. To achieve this, we firstly partition the input mesh with plane primitives, simplify it into a lightweight mesh next, then optimize plane parameters, camera poses and texture colors to maximize the photometric consistency across frame…
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Matterport, Inc.（Matterport公司）
We propose a novel approach to reconstruct RGB-D indoor scene based on plane primitives. Our approach takes as input a RGB-D sequence and a dense coarse mesh reconstructed from it, and generates a lightweight, low-polygonal mesh with clear face textures and sharp features without losing geometry details from the original scene. Compared to existing methods which only cover large planar regions in the scene, our method builds the entire scene by adaptive planes without losing geometry details and also preserves sharp features in the mesh. Experiments show that our method is more efficient to ge…
We present a novel neural surface reconstruction method called NeuralRoom for reconstructing room-sized indoor scenes directly from a set of 2D images. Recently, implicit neural representations have become a promising way to reconstruct surfaces from multiview images due to their high-quality results and simplicity. However, implicit neural representations usually cannot reconstruct indoor scenes well because they suffer severe shape-radiance ambiguity. We assume that the indoor scene consists of texture-rich and flat texture-less regions. In texture-rich regions, the multiview stereo can obta…
TwinTex: Geometry-aware Texture Generation for Abstracted 3D Architectural Models
Shenzhen University（深圳大学）；Guangdong Artificial Intelligence and Digital Economy Laboratory (SZ)（广东省人工智能与数字经济实验室（深圳））；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Recently unsupervised learning of depth from videos has made remarkable progress and the results are comparable to fully supervised methods in outdoor scenes like KITTI. However, there still exist great challenges when directly applying this technology in indoor environments, e.g., large areas of non-texture regions like white wall, more complex ego-motion of handheld camera, transparent glasses and shiny objects. To overcome these problems, we propose a new optical-flow based training paradigm which reduces the difficulty of unsupervised learning by providing a clearer training target and han…
Beijing University of Posts and Telecommunications（北京邮电大学）；Midea Group（美的集团）
The raw depth image captured by the indoor depth sensor usually has an extensive range of missing depth values due to inherent limitations such as the inability to perceive transparent objects and limited distance range. The incomplete depth map burdens many downstream vision tasks, and a rising number of depth completion methods have been proposed to alleviate this issue. While most existing methods can generate accurate dense depth maps from sparse and uniformly sampled depth maps, they are not suitable for complementing the large contiguous regions of missing depth values, which is common a…
Location-aware applications play an increasingly critical role in everyday life. However, satellite-based localization (e.g., GPS) has limited accuracy and can be unusable in dense urban areas and indoors. We introduce an image-based global localization system that is accurate to a few millimeters and performs reliable localization both indoors and outside. The key idea is to capture and index distinctive local keypoints in ground textures. This is based on the observation that ground textures including wood, carpet, tile, concrete, and asphalt may look random and homogeneous, but all contain …
This paper tackles the unsupervised depth estimation task in indoor environments. The task is extremely challenging because of the vast areas of non-texture regions in these scenes. These areas could overwhelm the optimization process in the commonly used unsupervised depth estimation framework proposed for outdoor environments. However, even when those regions are masked out, the performance is still unsatisfactory. In this paper, we argue that the poor performance suffers from the non-discriminative point-based matching. To this end, we propose P$^2$Net. We first extract points with large lo…
FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow
School of Computer Science, Peking University（北京大学计算机科学系）；Technical University of Munich（慕尼黑技术大学）；Beijing Jiaotong University（北京交通大学）；Beijing Digital Native Digital City Research Center（北京数字原生数字城市研究院）；Theta Labs, Inc.（Theta Labs公司）；Beijing Normal University（北京师范大学）
Robotics, TCS Research \& Innovation, Kolkata, India
Autonomous Micro Aerial Vehicles (MAVs) gained tremendous attention in recent years. Autonomous flight in indoor requires a dense depth map for navigable space detection which is the fundamental component for autonomous navigation. In this paper, we address the problem of reconstructing dense depth while a drone is hovering (small camera motion) in indoor scenes using already estimated cameras and sparse point cloud obtained from a vSLAM. We start by segmenting the scene based on sudden depth variation using sparse 3D points and introduce a patch-based local plane fitting via energy minimizati…
CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customization
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）
Department of Computer Science, George Mason University（乔治梅森大学计算机科学系）
The paper exploits weak Manhattan constraints to parse the structure of indoor environments from RGB-D video sequences in an online setting. We extend the previous approach for single view parsing of indoor scenes to video sequences and formulate the problem of recovering the floor plan of the environment as an optimal labeling problem solved using dynamic programming. The temporal continuity is enforced in a recursive setting, where labeling from previous frames is used as a prior term in the objective function. In addition to recovery of piecewise planar weak Manhattan structure of the exten…
Hefei University of Technology（合肥工业大学）；Orbbec（奥比中光）；Sun Yat-sen University（中山大学）
It is well known that the passive stereo system cannot adapt well to weak texture objects, e.g., white walls. However, these weak texture targets are very common in indoor environments. In this paper, we present a novel stereo system, which consists of two cameras (an RGB camera and an IR camera) and an IR speckle projector. The RGB camera is used both for depth estimation and texture acquisition. The IR camera and the speckle projector can form a monocular structured-light (MSL) subsystem, while the two cameras can form a binocular stereo subsystem. The depth map generated by the MSL subsyste…
This paper presents a novel indoor layout estimation system based on the fusion of 2D LiDAR and intensity camera data. A ground robot explores an indoor space with a single floor and vertical walls, and collects a sequence of intensity images and 2D LiDAR datasets. The LiDAR provides accurate depth information, while the camera captures high-resolution data for semantic interpretation. The alignment of sensor outputs and image segmentation are computed jointly by aligning LiDAR points, as samples of the room contour, to ground-wall boundaries in the images. The alignment problem is decoupled i…
Simultaneous mapping and localization (SLAM) in an real indoor environment is still a challenging task. Traditional SLAM approaches rely heavily on low-level geometric constraints like corners or lines, which may lead to tracking failure in textureless surroundings or cluttered world with dynamic objects. In this paper, a compact semantic SLAM framework is proposed, with utilization of both geometric and object-level semantic constraints jointly, a more consistent mapping result, and more accurate pose estimation can be obtained. Two main contributions are presented int the paper, a) a robust …
Physically Realistic Sequence-Level Adversarial Clothing for Robust Human-Detection Evasion
School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）；Institute for Interdisciplinary Information Sciences, Tsinghua University（交叉信息研究院，清华大学）
