Interactive visual grounding in Human-Robot Interaction (HRI) is challenging yet practical due to the inevitable ambiguity in natural languages. It requires robots to disambiguate the user input by active information gathering. Previous approaches often rely on predefined templates to ask disambiguation questions, resulting in performance reduction in realistic interactive scenarios. In this paper, we propose TiO, an end-to-end system for interactive visual grounding in human-robot interaction. Benefiting from a unified formulation of visual dialogue and grounding, our method can be trained on a joint of extensive public data, and show superior generality to diversified and challenging open-world scenarios. In the experiments, we validate TiO on GuessWhat?! and InViG benchmarks, setting new state-of-the-art performance by a clear margin. Moreover, we conduct HRI experiments on the carefully selected 150 challenging scenes as well as real-robot platforms. Results show that our method demonstrates superior generality to diversified visual and language inputs with a high success rate. Codes and demos are available at https://github.com/jxu124/TiO.
Deploying interactive systems in-the-wild requires adaptability to situations not encountered in lab environments. Our work details our experience about the impact of architecture choice on behavior reusability and reactivity while deploying a public interactive system. In particular, we introduce Shutter, a robot photographer and a platform for public interaction. In designing Shutter's architecture, we focused on adaptability for in-the-wild deployment, while developing a reusable platform to facilitate future research in public human-robot interaction. We find that behavior trees allow reac…
This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions. INGRESS allows for unconstrained object categories and unconstrained language expressions. Further, it asks questions to disambiguate referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural network model for grounding. The first stage uses a neural…
Are We Generalizing from the Exception? An In-the-Wild Study on Group-Sensitive Conversation Design in Human-Agent Interactions
Cologne Cobots Lab, TH Köln - University of Applied Sciences（科隆机器人实验室，TH Köln应用科学大学）；KI Park e.V. & FAU - Friedrich-Alexander University of Erlangen-Nuremberg（KI Park协会与弗赖堡-艾森纳赫大学）
Classification of User Satisfaction in HRI with Social Signals in the Wild
TH Köln - University of Applied Sciences, Cologne Cobots Lab, Betzdorfer Str. 2, 50679, Germany（TH Köln应用科学大学，科隆Cobots实验室）；KI Park e.V. & FAU - Friedrich-Alexander University of Erlangen-Nuremberg（KI Park协会与弗赖堡-埃尔兰根-纽伦堡大学）
SAIN: Structure-Aware Interactive Navigation with Active Dialogue Grounding for Mobile Robot
School of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）机械工程与自动化学院）；Department of Mechanical Engineering, City University of Hong Kong（香港城市大学机械工程系）
Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of t…
Runtime Learning of Quadruped Robots in Wild Environments
Engineering Technology Division, Wayne State University（韦恩州立大学工程技术系）；Department of Computer Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；School of Engineering and Design, Technical University of Munich（慕尼黑技术大学工程与设计学院）；School of Electrical Engineering and Computer Science, Washington State University（华盛顿州立大学电气与计算机科学学院）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the…
