The problem of tracking self-motion as well as motion of objects in the scene using information from a camera is known as multi-body visual odometry and is a challenging task. This paper proposes a robust solution to achieve accurate estimation and consistent track-ability for dynamic multi-body visual odometry. A compact and effective framework is proposed leveraging recent advances in semantic instance-level segmentation and accurate optical flow estimation. A novel formulation, jointly optimizing SE(3) motion and optical flow is introduced that improves the quality of the tracked points and the motion estimation accuracy. The proposed approach is evaluated on the virtual KITTI Dataset and tested on the real KITTI Dataset, demonstrating its applicability to autonomous driving applications. For the benefit of the community, we make the source code public.
This paper presents two visual trackers from the different paradigms of learning and registration based tracking and evaluates their application in image based visual servoing. They can track object motion with four degrees of freedom (DoF) which, as we will show here, is sufficient for many fine manipulation tasks. One of these trackers is a newly developed learning based tracker that relies on learning discriminative correlation filters while the other is a refinement of a recent 8 DoF RANSAC based tracker adapted with a new appearance model for tracking 4 DoF motion. Both trackers are shown…
Hong Kong University of Science and Technology（香港科技大学）
We propose a stereo vision-based approach for tracking the camera ego-motion and 3D semantic objects in dynamic autonomous driving scenarios. Instead of directly regressing the 3D bounding box using end-to-end approaches, we propose to use the easy-to-labeled 2D detection and discrete viewpoint classification together with a light-weight semantic inference method to obtain rough 3D object measurements. Based on the object-aware-aided camera pose tracking which is robust in dynamic environments, in combination with our novel dynamic object bundle adjustment (BA) approach to fuse temporal sparse…
Ego-pose estimation and dynamic object tracking are two key issues in an autonomous driving system. Two assumptions are often made for them, i.e. the static world assumption of simultaneous localization and mapping (SLAM) and the exact ego-pose assumption of object tracking, respectively. However, these assumptions are difficult to hold in highly dynamic road scenarios where SLAM and object tracking become correlated and mutually beneficial. In this paper, DL-SLOT, a dynamic Lidar SLAM and object tracking method is proposed. This method integrates the state estimations of both the ego vehicle …
Incorporating the physical environment is essential for a complete understanding of human behavior in unconstrained every-day tasks. This is especially important in ego-centric tasks where obtaining 3 dimensional information is both limiting and challenging with the current 2D video analysis methods proving insufficient. Here we demonstrate a proof-of-concept system which provides real-time 3D mapping and semantic labeling of the local environment from an ego-centric RGB-D video-stream with 3D gaze point estimation from head mounted eye tracking glasses. We augment existing work in Semantic Si…
EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Observations
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The University of Hong Kong（香港大学）；Miraikan – The National Museum of Emerging Science and Innovation（Miraikan——新兴科学与创新国家博物馆）
University of Electronic Science and Technology of China（电子科技大学）；Tsinghua University（清华大学）
Localization in a dynamic environment suffers from moving objects. Removing dynamic object is crucial in this situation but become tricky when ego-motion is coupled. In this paper, instead of proposing a new slam framework, we aim at a more general strategy for a localization scenario. In that case, Dynamic Registration is available for integrating with any lidar slam system. We utilize 3D object detection to obtain potential moving objects and remove them temporarily. Then we proposed Dynamic Registration, to iteratively estimate ego-motion and segment moving objects until no static object ge…
LIMOT: A Tightly-Coupled System for LiDAR-Inertial Odometry and Multi-Object Tracking
Tongji University（同济大学）；Institute of Intelligent Vehicles, Tongji University（同济大学智能车辆研究所）；School of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；School of Surveying and Geo-Informatics, Tongji University（同济大学测绘与地理信息学院）
Concurrently estimating the 6-DOF pose of multiple cameras or robots---cooperative localization---is a core problem in contemporary robotics. Current works focus on a set of mutually observable world landmarks and often require inbuilt egomotion estimates; situations in which both assumptions are violated often arise, for example, robots with erroneous low quality odometry and IMU exploring an unknown environment. In contrast to these existing works in cooperative localization, we propose a cooperative localization method, which we call mutual localization, that uses reciprocal observations of…
Keypoint-based Dynamic Object 6-DoF Pose Tracking via Event Camera
ShanghaiTech University（上海科技大学）；ShanghaiTech Automation and Robotics (STAR) Center（上海科技大学自动化与机器人中心）；School of Information Science and Technology（信息科学与技术学院）；WLSA Shanghai Academy（WLSA上海研究院）
Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high-speed motions or in scenes characterized by high dynamic range. These features, along with a very low power consumption, make event cameras an ideal complement to standard cameras for VR/AR and video game applications. With these applications in mind, this paper tackles the problem of accurate, low-latency tracking of an…
Human-as-Humanoid: Enabling Zero-Shot Humanoid Learning from Ego-Exo Human Videos with Human-Aligned Embodiments
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；DeepCybo；ZGCA；ZGCI；Harbin Institute of Technology（哈尔滨工业大学）；Huazhong University of Science and Technology（华中科技大学）；Beihang University（北京航空航天大学）
Position sensitive detectors (PSDs) offer possibility to track single active marker's two (or three) degrees of freedom (DoF) position with a high accuracy, while having a fast response time with high update frequency and low latency, all using a very simple signal processing circuit. However they are not particularly suitable for 6-DoF object pose tracking system due to lack of orientation measurement, limited tracking range, and sensitivity to environmental variation. We propose a novel 6-DoF pose tracking system for a rigid object tracking requiring a single active marker. The proposed syst…
University of Stuttgart（斯图加特大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；TU Berlin（柏林工业大学）
Robotic manipulation of unknown objects is an important field of research. Practical applications occur in many real-world settings where robots need to interact with an unknown environment. We tackle the problem of reactive grasping by proposing a method for unknown object tracking, grasp point sampling and dynamic trajectory planning. Our object tracking method combines Siamese Networks with an Iterative Closest Point approach for pointcloud registration into a method for 6-DoF unknown object tracking. The method does not require further training and is robust to noise and occlusion. We prop…
InvariantCloud: A Globally Invariant, Uniquely Indexed Point Cloud Framework for Robust 6-DoF Tactile Pose Tracking
Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学机械与航空航天工程系）；Department of Mechanical Engineering, Massachusetts Institute of Technology（麻省理工学院机械工程系）；Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong（香港中文大学机械与自动化工程系）
Tensegrity robots, which are composed of compressive elements (rods) and flexible tensile elements (e.g., cables), have a variety of advantages, including flexibility, low weight, and resistance to mechanical impact. Nevertheless, the hybrid soft-rigid nature of these robots also complicates the ability to localize and track their state. This work aims to address what has been recognized as a grand challenge in this domain, i.e., the state estimation of tensegrity robots through a markerless, vision-based method, as well as novel, onboard sensors that can measure the length of the robot's cabl…
Face versus Body Tracking for Human-Robot Interaction: An Egocentric Dataset
Furhat Robotics；University of Naples Federico II（那不勒斯费德里科二世大学）；Division of Speech, Music and Hearing, KTH Royal Institute of Technology（语音、音乐和听觉研究所，皇家理工学院）
Maximum Solar Energy Tracking Leverage High-DoF Robotics System with Deep Reinforcement Learning
North Carolina State University（北卡罗来纳州立大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Kyoto University（京都大学）；Monash University（莫纳什大学）；Indian Institute of Technology（印度理工学院）；South East European University（东南欧大学）；University of Buenos Aires（布宜诺斯艾利斯大学）
We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence, while simultaneously performing neural 3D reconstruction of the object. Our method works for arbitrary rigid objects, even when visual texture is largely absent. The object is assumed to be segmented in the first frame only. No additional information is required, and no assumption is made about the interaction agent. Key to our method is a Neural Object Field that is learned concurrently with a pose graph optimization process in order to robustly accumulate information into a consis…
EgoPriMo: Egocentric Motion Generation for Interactive Humanoid Control
Tianjin University（天津大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；DeepCybo
Enhanced Real-Time 6-DOF Extended Reality Catheter Tracking for Evaluating Potential Improvement in Efficiency, Precision, and Depth Perception for Cardiac Interventions
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）
We propose a single-stage, category-level 6-DoF pose estimation algorithm that simultaneously detects and tracks instances of objects within a known category. Our method takes as input the previous and current frame from a monocular RGB video, as well as predictions from the previous frame, to predict the bounding cuboid and 6-DoF pose (up to scale). Internally, a deep network predicts distributions over object keypoints (vertices of the bounding cuboid) in image coordinates, after which a novel probabilistic filtering process integrates across estimates before computing the final pose using P…
You Only Scan Once: A Dynamic Scene Reconstruction Pipeline for 6-DoF Robotic Grasping of Novel Objects
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Integrative Sciences and Engineering Programme, National University of Singapore Graduate School（新加坡国立大学研究生院整合科学与工程项目）
ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；TeleAI, China Telecom（TeleAI，中国电信）
Many practical applications of robotics require systems that can operate safely despite uncertainty. In the context of motion planning, two types of uncertainty are particularly important when planning safe robot trajectories. The first is environmental uncertainty -- uncertainty in the locations of nearby obstacles, stemming from sensor noise or (in the case of obstacles' future locations) prediction error. The second class of uncertainty is uncertainty in the robots own state, typically caused by tracking or estimation error. To achieve high levels of safety, it is necessary for robots to co…
A Unidirectionally Connected FAS Approach for 6-DOF Quadrotor Control
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology（广东省全驱动系统控制理论与技术重点实验室）；Southern University of Science and Technology（南方科技大学）；Center for Control Theory and Guidance Technology（控制理论与导航技术中心）；Harbin Institute of Technology（哈尔滨工业大学）
Advanced XR-Based 6-DOF Catheter Tracking System for Immersive Cardiac Intervention Training
Dalio Institute of Cardiovascular Imaging（达利奥心血管成像研究所）；Weill Cornell Medicine（威尔康奈尔医学院）；Englander Institute for Precision Medicine（英格兰德精准医学研究所）；Institute for Computational Biomedicine（计算生物医学研究所）
Inertial information processing plays a pivotal role in ego-motion awareness for mobile agents, as inertial measurements are entirely egocentric and not environment dependent. However, they are affected greatly by changes in sensor placement/orientation or motion dynamics, and it is infeasible to collect labelled data from every domain. To overcome the challenges of domain adaptation on long sensory sequences, we propose a novel framework that extracts domain-invariant features of raw sequences from arbitrary domains, and transforms to new domains without any paired data. Through the experimen…
Dynamic Properties and Motion Reproducibility of a Compact Pneumatically Actuated Humanoid Upper Body for Data-Driven Control
Symbiotic Intelligent Systems Research Center, Institute for Open and Transdisciplinary Research Initiatives, The University of Osaka（共生智能系统研究中心，开放与跨学科研究倡议研究所，大阪大学）；International Professional University of Technology in Osaka, Umeda, Kita-ku, Osaka, Japan（大阪国际专业技术大学，乌梅达，北区，大阪，日本）；Chubu University Academy of Emerging Sciences, Kasugai, Aichi, Japan（名古屋大学新兴科学学院， Kasugai，爱知，日本）
Design, Dynamic Modeling and Control of a 2-DOF Robotic Wrist Actuated by Twisted and Coiled Actuators
College of Advanced Manufacturing and Robotics（先进制造与机器人学院）；State Key Laboratory of Turbulence and Complex Systems（湍流与复杂系统国家重点实验室）；Peking University（北京大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）
