Computer Vision and Geometry Group, ETH Zurich（苏黎世联邦理工学院计算机视觉与几何组）；Mobile Robotics Lab, Technical University of Munich（慕尼黑技术大学移动机器人实验室）；Robotic Systems Lab, ETH Zurich（苏黎世联邦理工学院机器人系统实验室）；Microsoft Mixed Reality and AI Lab（微软混合现实与人工智能实验室）；Robot Perception and Learning Lab, University of Bonn（波恩大学机器人感知与学习实验室）
Boyang Sun, Hanzhi Chen, Stefan Leutenegger, Cesar Cadena, Marc Pollefeys, Hermann Blum
Exploration of unknown environments is crucial for autonomous robots; it allows them to actively reason and decide on what new data to acquire for different tasks, such as mapping, object discovery, and environmental assessment. Existing solutions, such as frontier-based exploration approaches, rely heavily on 3D map operations, which are limited by map quality and, more critically, often overlook valuable context from visual cues. This work aims at leveraging 2D visual cues for efficient autonomous exploration, addressing the limitations of extracting goal poses from a 3D map. We propose a visual-only frontier-based exploration system, with FrontierNet as its core component. FrontierNet is a learning-based model that (i) proposes frontiers, and (ii) predicts their information gain, from posed RGB images enhanced by monocular depth priors. Our approach provides an alternative to existing 3D-dependent goal-extraction approaches, achieving a 15\% improvement in early-stage exploration efficiency, as validated through extensive simulations and real-world experiments. The project is available at https://github.com/cvg/FrontierNet.
In this paper, we propose to augment image-based place recognition with structural cues. Specifically, these structural cues are obtained using structure-from-motion, such that no additional sensors are needed for place recognition. This is achieved by augmenting the 2D convolutional neural network (CNN) typically used for image-based place recognition with a 3D CNN that takes as input a voxel grid derived from the structure-from-motion point cloud. We evaluate different methods for fusing the 2D and 3D features and obtain best performance with global average pooling and simple concatenation. …
Monta Vista High School（蒙塔维斯塔高中）；Stanford University（斯坦福大学）
We propose a K-Means based prediction system, nicknamed SERVANT (Scene Recognition Through Visual and Acoustic Cues), that is capable of recognizing environmental scenes through analysis of ambient sound and color cues. The concept and implementation originated within the Learning branch of the Intelligent Wearable Robotics Project (also known as the Third Arm project) at the Stanford Artificial Intelligence Lab-Toyota Center (SAIL-TC). The Third Arm Project focuses on the development and conceptualization of a robotic arm that can aid users in a whole array of situations: i.e. carrying a cup …
Indian Institute of Science（印度科学学院）；Robert Bosch Center for Cyber-Physical Systems（罗伯特·博世信息物理系统研究中心）
A robot that can operate in novel and unstructured environments must be capable of recognizing new, previously unseen, objects. In this work, a visual cue is used to specify a novel object of interest which must be localized in new environments. An end-to-end neural network equipped with a Siamese network is used to learn the cue, infer the object of interest, and then to localize it in new environments. We show that a simulated robot can pick-and-place novel objects pointed to by a laser pointer. We also evaluate the performance of the proposed approach on a dataset derived from the Omniglot …
The gap between simulation and the real-world restrains many machine learning breakthroughs in computer vision and reinforcement learning from being applicable in the real world. In this work, we tackle this gap for the specific case of camera-based navigation, formulating it as following a visual cue in the foreground with arbitrary backgrounds. The visual cue in the foreground can often be simulated realistically, such as a line, gate or cone. The challenge then lies in coping with the unknown backgrounds and integrating both. As such, the goal is to train a visual agent on data captured in …
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）；INRIA（法国国家信息与自动化研究所）
This paper addresses the problem of building augmented metric representations of scenes with semantic information from RGB-D images. We propose a complete framework to create an enhanced map representation of the environment with object-level information to be used in several applications such as human-robot interaction, assistive robotics, visual navigation, or in manipulation tasks. Our formulation leverages a CNN-based object detector (Yolo) with a 3D model-based segmentation technique to perform instance semantic segmentation, and to localize, identify, and track different classes of objec…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos don't come with labels for actions or goals, and may not even showcase optimal behavior. Our method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning …
School of CS & AI, Arizona State University（亚利桑那州立大学计算机科学与人工智能学院）
Conventional works that learn grasping affordance from demonstrations need to explicitly predict grasping configurations, such as gripper approaching angles or grasping preshapes. Classic motion planners could then sample trajectories by using such predicted configurations. In this work, our goal is instead to fill the gap between affordance discovery and affordance-based policy learning by integrating the two objectives in an end-to-end imitation learning framework based on deep neural networks. From a psychological perspective, there is a close association between attention and affordance. T…
Rapid Autonomous Car Control based on Spatial and Temporal Visual Cues
Monta Vista High School（蒙塔维斯塔高中）
我们提出了一种利用深度卷积神经网络和长短期记忆系统相结合的现代汽车控制的新方法：这两者都是分层表示学习的一个子部分，通常被称为深度学习。使用深度卷积神经网络和长短期记忆系统（DCNN/LSTM），我们提出了一种端到端的方法来准确预测转向角和油门值。我们在最新的机器人El Toro Grande 1（ETG）上使用这种算法，该机器人配备了各种传感器，以便在其环境中定位自己。使用以前的训练数据和它在电路和阻力比赛中收集的数据，它预测油门和转向角度，以保持在路径上，避免碰撞到其他机器人。这使得ETG理论上可以在任何有足够训练数据的赛道上比赛。
