We present iGibson 1.0, a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains 15 fully interactive home-sized scenes with 108 rooms populated with rigid and articulated objects. The scenes are replicas of real-world homes, with distribution and the layout of objects aligned to those of the real world. iGibson 1.0 integrates several key features to facilitate the study of interactive tasks: i) generation of high-quality virtual sensor signals (RGB, depth, segmentation, LiDAR, flow and so on), ii) domain randomization to change the materials of the objects (both visual and physical) and/or their shapes, iii) integrated sampling-based motion planners to generate collision-free trajectories for robot bases and arms, and iv) intuitive human-iGibson interface that enables efficient collection of human demonstrations. Through experiments, we show that the full interactivity of the scenes enables agents to learn useful visual representations that accelerate the training of downstream manipulation tasks. We also show that iGibson 1.0 features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of human demonstrated (mobile) manipulation behaviors. iGibson 1.0 is open-source, equipped with comprehensive examples and documentation. For more information, visit our project website: http://svl.stanford.edu/igibson/
Recent research in embodied AI has been boosted by the use of simulation environments to develop and train robot learning approaches. However, the use of simulation has skewed the attention to tasks that only require what robotics simulators can simulate: motion and physical contact. We present iGibson 2.0, an open-source simulation environment that supports the simulation of a more diverse set of household tasks through three key innovations. First, iGibson 2.0 supports object states, including temperature, wetness level, cleanliness level, and toggled and sliced states, necessary to cover a …
Stanford University（斯坦福大学）；Robotics at Google（谷歌机器人部门）
We present Interactive Gibson Benchmark, the first comprehensive benchmark for training and evaluating Interactive Navigation: robot navigation strategies where physical interaction with objects is allowed and even encouraged to accomplish a task. For example, the robot can move objects if needed in order to clear a path leading to the goal location. Our benchmark comprises two novel elements: 1) a new experimental setup, the Interactive Gibson Environment (iGibson 0.5), which simulates high fidelity visuals of indoor scenes, and high fidelity physical dynamics of the robot and common objects …
UCLA Center for Vision, Cognition, Learning, and Autonomy (VCLA)（加州大学洛杉矶分校视觉、认知、学习与自主中心（VCLA））
In this paper, we rethink the problem of scene reconstruction from an embodied agent's perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide \em{actionable} information for simulating \em{interactions} with agents. Here, we address this challenging problem by reconstructing an interactive scene using RGB-D data stream, which captures (i) the semantics and geometry of objects and layouts by a 3D volumetric panoptic mapping module, and (ii) object affordance an…
Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objects in everyday life, this work explores building articulation models of indoor scenes through a robot's purposeful interactions in these scenes. Prior work on articulation reasoning primarily focuses on siloed objects of limited categories. To extend to room-scale environments, the robot has to efficiently and effectively explore a large-scale 3D space, locate articulated objects, and infer their articulations. We in…
SceneFoundry: Generating Interactive Infinite 3D Worlds
National Yang Ming Chiao Tung University（国家阳明交通大学）；National Tsing Hua University（国立清华大学）；NVIDIA AI Technology Center（NVIDIA AI技术中心）；National Taiwan University（国立台湾大学）
RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Columbia University（哥伦比亚大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon（亚马逊公司）
Recent years have witnessed some exciting developments in the domain of generating images from scene-based text descriptions. These approaches have primarily focused on generating images from a static text description and are limited to generating images in a single pass. They are unable to generate an image interactively based on an incrementally additive text description (something that is more intuitive and similar to the way we describe an image). We propose a method to generate an image incrementally based on a sequence of graphs of scene descriptions (scene-graphs). We propose a recurren…
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院（BIGAI））
Tsinghua University（清华大学）；Massachusetts Institute of Technology（麻省理工学院）
Predicting future motions of road participants is an important task for driving autonomously in urban scenes. Existing models excel at predicting marginal trajectories for single agents, yet it remains an open question to jointly predict scene compliant trajectories over multiple agents. The challenge is due to exponentially increasing prediction space as a function of the number of agents. In this work, we exploit the underlying relations between interacting agents and decouple the joint prediction problem into marginal prediction problems. Our proposed approach M2I first classifies interacti…
Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks
College of Electronic and Information Engineering, Tongji University（同济大学电子信息学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统研究所）；College of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；Department of Vehicle Control System and Software Development, NIO（蔚来汽车车辆控制系统与软件开发部门）；Department of Automotive Engineering, Jilin University（吉林大学汽车工程学院）；Key Laboratory of Embedded System and Service Computing (Ministry of Education), Tongji University（同济大学嵌入式系统与服务计算重点实验室）；College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology（上海智能科学与技术研究院电子信息学院）；Shanghai Key Laboratory of Intelligent Autonomous Systems（上海智能自主系统重点实验室）
Unravel: Recovering an Interactive 3D Scene from a Single Image
Zhejiang University（浙江大学）；Shanghai Innovation Institute（上海创新研究院）；Westlake University（西湖大学）；LIGHTSPEED；Adobe（奥多比公司）；Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）；University of Hong Kong（香港大学）
Building embodied intelligent agents that can interact with 3D indoor environments has received increasing research attention in recent years. While most works focus on single-object or agent-object visual functionality and affordances, our work proposes to study a new kind of visual relationship that is also important to perceive and model -- inter-object functional relationships (e.g., a switch on the wall turns on or off the light, a remote control operates the TV). Humans often spend little or no effort to infer these relationships, even when entering a new room, by using our strong prior …
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Key Laboratory of General Artificial Intelligence, BIGAI（北京智源人工智能研究院通用人工智能国家重点实验室）；Department of Automation, Tsinghua University（清华大学自动化系）
To solve its task, a robot needs to have the ability to interpret its perceptions. In vision, this interpretation is particularly difficult and relies on the understanding of the structure of the scene, at least to the extent of its task and sensorimotor abilities. A robot with the ability to build and adapt this interpretation process according to its own tasks and capabilities would push away the limits of what robots can achieve in a non controlled environment. A solution is to provide the robot with processes to build such representations that are not specific to an environment or a situat…
Robots need to understand their environment to perform their task. If it is possible to pre-program a visual scene analysis process in closed environments, robots operating in an open environment would benefit from the ability to learn it through their interaction with their environment. This ability furthermore opens the way to the acquisition of affordances maps in which the action capabilities of the robot structure its visual scene understanding. We propose an approach to build such affordances maps by relying on an interactive perception approach and an online classification. In the propo…
Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；Stanford University（斯坦福大学）；Carnegie Mellon University（卡内基梅隆大学）
We introduce The House Of inteRactions (THOR), a framework for visual AI research, available at http://ai2thor.allenai.org. AI2-THOR consists of near photo-realistic 3D indoor scenes, where AI agents can navigate in the scenes and interact with objects to perform tasks. AI2-THOR enables research in many different domains including but not limited to deep reinforcement learning, imitation learning, learning by interaction, planning, visual question answering, unsupervised representation learning, object detection and segmentation, and learning models of cognition. The goal of AI2-THOR is to fac…
TU Wien（维也纳技术大学）；Institute of Science and Technology Austria (IST Austria)（奥地利科学技术研究所）；University of Tübingen（蒂宾根大学）；Massachusetts Institute of Technology (MIT)（麻省理工学院）
While convolutional neural networks (CNNs) have found wide adoption as state-of-the-art models for image-related tasks, their predictions are often highly sensitive to small input perturbations, which the human vision is robust against. This paper presents Perturber, a web-based application that allows users to instantaneously explore how CNN activations and predictions evolve when a 3D input scene is interactively perturbed. Perturber offers a large variety of scene modifications, such as camera controls, lighting and shading effects, background modifications, object morphing, as well as adve…
Indian Institute of Technology, Madras（马德拉斯印度理工学院）；IBM Research Labs, Bangalore（IBM研究院班加罗尔实验室）；Indian Institute of Technology, Bombay（孟买印度理工学院）
With the prolification of multimodal interaction in various domains, recently there has been much interest in text based image retrieval in the computer vision community. However most of the state of the art techniques model this problem in a purely neural way, which makes it difficult to incorporate pragmatic strategies in searching a large scale catalog especially when the search requirements are insufficient and the model needs to resort to an interactive retrieval process through multiple iterations of question-answering. Motivated by this, we propose a neural-symbolic approach for a one-s…
S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering
University of Science and Technology of China（中国科学技术大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；City University of Hong Kong（香港城市大学）；The Chinese University of Hong Kong（香港中文大学）；Fudan University（复旦大学）；The University of Sydney（悉尼大学）；Beihang University（北航）
