Localizing objects and estimating their extent in 3D is an important step towards high-level 3D scene understanding, which has many applications in Augmented Reality and Robotics. We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN). Based on these frame-to-model associations, our back-end optimizes object bounding volumes, represented as super-quadrics, under multi-view geometry constraints and the object scale prior. We validate the proposed system on ScanNet where we show a significant improvement over existing RGB-only methods.
Cross-modal State Space Modeling for Real-time RGB-thermal Wild Scene Semantic Segmentation
School of Automation, Beijing Institute of Technology（自动化学院，北京理工大学）；School of Information and Electronic Engineering, Zhejiang University of Science and Technology（信息电子工程学院，浙江工业大学）；School of Computer Science and Engineering, Nanyang Technological University（计算机科学与工程学院，南洋理工大学）
Simultaneous mapping and localization (SLAM) in an real indoor environment is still a challenging task. Traditional SLAM approaches rely heavily on low-level geometric constraints like corners or lines, which may lead to tracking failure in textureless surroundings or cluttered world with dynamic objects. In this paper, a compact semantic SLAM framework is proposed, with utilization of both geometric and object-level semantic constraints jointly, a more consistent mapping result, and more accurate pose estimation can be obtained. Two main contributions are presented int the paper, a) a robust …
Towards Localizing Structural Elements: Merging Geometrical Detection with Semantic Verification in RGB-D Data
University of Luxembourg（卢森堡大学）；Interdisciplinary Centre for Security, Reliability, and Trust (SnT)（安全、可靠性与跨学科中心（SnT））；Institute for Advanced Studies (IAS)（高等研究院（IAS））；Faculty of Science, Technology, and Medicine（科学、技术与医学院）
Traditional SLAM algorithms are typically based on artificial features, which lack high-level information. By introducing semantic information, SLAM can own higher stability and robustness rather than purely hand-crafted features. However, the high uncertainty of semantic detection networks prohibits the practical functionality of high-level information. To solve the uncertainty property introduced by semantics, this paper proposed a novel probability map based on the Gaussian distribution assumption. This map transforms the semantic binary object detection into probability results, which help…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
The majority of approaches for acquiring dense 3D environment maps with RGB-D cameras assumes static environments or rejects moving objects as outliers. The representation and tracking of moving objects, however, has significant potential for applications in robotics or augmented reality. In this paper, we propose a novel approach to dynamic SLAM with dense object-level representations. We represent rigid objects in local volumetric signed distance function (SDF) maps, and formulate multi-object tracking as direct alignment of RGB-D images with the SDF representations. Our main novelty is a pr…
We present an interesting and challenging dataset that features a large number of scenes with messy tables captured from multiple camera views. Each scene in this dataset is highly complex, containing multiple object instances that could be identical, stacked and occluded by other instances. The key challenge is to associate all instances given the RGB image of all views. The seemingly simple task surprisingly fails many popular methods or heuristics that we assume good performance in object association. The dataset challenges existing methods in mining subtle appearance differences, reasoning…
University of Adelaide（阿德莱德大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Monash University（莫纳什大学）
Semantic aware reconstruction is more advantageous than geometric-only reconstruction for future robotic and AR/VR applications because it represents not only where things are, but also what things are. Object-centric mapping is a task to build an object-level reconstruction where objects are separate and meaningful entities that convey both geometry and semantic information. In this paper, we present MOLTR, a solution to object-centric mapping using only monocular image sequences and camera poses. It is able to localise, track, and reconstruct multiple objects in an online fashion when an RGB…
Imperial College London（伦敦帝国学院）；Technical University of Munich（慕尼黑工业大学）
In this paper, we propose a novel object-level mapping system that can simultaneously segment, track, and reconstruct objects in dynamic scenes. It can further predict and complete their full geometries by conditioning on reconstructions from depth inputs and a category-level shape prior with the aim that completed object geometry leads to better object reconstruction and tracking accuracy. For each incoming RGB-D frame, we perform instance segmentation to detect objects and build data associations between the detection and the existing object maps. A new object map will be created for each un…
Autonomous systems need to localize and track surrounding objects in 3D space for safe motion planning. As a result, 3D multi-object tracking (MOT) plays a vital role in autonomous navigation. Most MOT methods use a tracking-by-detection pipeline, which includes object detection and data association processing. However, many approaches detect objects in 2D RGB sequences for tracking, which is lack of reliability when localizing objects in 3D space. Furthermore, it is still challenging to learn discriminative features for temporally-consistent detection in different frames, and the affinity mat…
TENet: Targetness Entanglement Incorporating with Multi-Scale Pooling and Mutually-Guided Fusion for RGB-E Object Tracking
School of Artificial Intelligence and Computer Science, Jiangnan University（江南大学人工智能与计算机科学学院）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）
SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos
College of Computer Science and Engineering, Guilin University of Technology（桂林理工大学计算机科学与工程学院）；Guangxi Key Laboratory of Embedded Technology and Intelligent System, Guilin University of Technology（广西嵌入式技术与智能系统重点实验室）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；School of Electronic Information, Central South University（中南大学电子信息学院）
Georgia Institute of Technology（佐治亚理工学院）；Oregon State University（俄勒冈州立大学）；University of California, San Diego（加州大学圣地亚哥分校）
We address the problems of measuring geometric similarity between 3D scenes, represented through point clouds or range data frames, and associating them. Our approach leverages macro-scale 3D structural geometry - the relative configuration of arbitrary surfaces and relationships among structures that are potentially far apart. We express such discriminative information in a viewpoint-invariant feature space. These are subsequently encoded in a frame-level signature that can be utilized to measure geometric similarity. Such a characterization is robust to noise, incomplete and partially overla…
Department of Computer Science（计算机科学系）；Robotics Institute（机器人研究所）；Toyota Research Institute（丰田研究机构）；Faculty of Science and Technology（科学与技术学院）；Institute of Collaborative Innovation（协同创新研究所）
DEGS: Deformable Event-based 3D Gaussian Splatting from RGB and Event Stream
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；University of Hong Kong（香港大学）；Northeastern University（东北大学）
Towards Universal Modal Tracking with Online Dense Temporal Token Learning
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education（教育区块链与智能技术重点实验室，教育部）；Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University（广西多源信息挖掘与安全重点实验室，广西师范大学）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；University of Chinese Academy of Sciences（中国科学院大学）；Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University（媒体分析与计算实验室，人工智能系，厦门大学）
FEMOT: Multi-Object Tracking using Frame and Event Cameras
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；Peng Cheng Laboratory（鹏城实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理全国重点实验室）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
Unsupervised video object segmentation aims to segment a target object in the video without a ground truth mask in the initial frame. This challenging task requires extracting features for the most salient common objects within a video sequence. This difficulty can be solved by using motion information such as optical flow, but using only the information between adjacent frames results in poor connectivity between distant frames and poor performance. To solve this problem, we propose a novel prototype memory network architecture. The proposed model effectively extracts the RGB and motion infor…
Active Adversarial Perturbation-driven Associative Memory Retrieval for RGB-Event Visual Object Tracking
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；School of Electronic and Information Engineering, Anhui University（安徽大学电子信息工程学院）；Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；School of Computer Science, Peking University（北京大学计算机学院）；School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）
Recent online Multi-Object Tracking (MOT) methods have achieved desirable tracking performance. However, the tracking speed of most existing methods is rather slow. Inspired from the fact that the adjacent frames are highly relevant and redundant, we divide the frames into key and non-key frames respectively and track objects in the compressed domain. For the key frames, the RGB images are restored for detection and data association. To make data association more reliable, an appearance Convolutional Neural Network (CNN) which can be jointly trained with the detector is proposed. For the non-k…
Stanford Vision and Learning Laboratory, Stanford University（斯坦福大学视觉与学习实验室）
Robots navigating autonomously need to perceive and track the motion of objects and other agents in its surroundings. This information enables planning and executing robust and safe trajectories. To facilitate these processes, the motion should be perceived in 3D Cartesian space. However, most recent multi-object tracking (MOT) research has focused on tracking people and moving objects in 2D RGB video sequences. In this work we present JRMOT, a novel 3D MOT system that integrates information from RGB images and 3D point clouds to achieve real-time, state-of-the-art tracking performance. Our sy…
We present a fast, scalable, and accurate Simultaneous Localization and Mapping (SLAM) system that represents indoor scenes as a graph of objects. Leveraging the observation that artificial environments are structured and occupied by recognizable objects, we show that a compositional scalable object mapping formulation is amenable to a robust SLAM solution for drift-free large scale indoor reconstruction. To achieve this, we propose a novel semantically assisted data association strategy that obtains unambiguous persistent object landmarks, and a 2.5D compositional rendering method that enable…
Autonomous Systems Lab, ETH Zurich（苏黎世联邦理工学院自主系统实验室）
To autonomously navigate and plan interactions in real-world environments, robots require the ability to robustly perceive and map complex, unstructured surrounding scenes. Besides building an internal representation of the observed scene geometry, the key insight toward a truly functional understanding of the environment is the usage of higher-level entities during mapping, such as individual object instances. We propose an approach to incrementally build volumetric object-centric maps during online scanning with a localized RGB-D camera. First, a per-frame segmentation scheme combines an uns…
