MQA: Answering the Question via Robotic Manipulation
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tsinghua University（清华大学）
Yuhong Deng, Di Guo, Xiaofeng Guo, Naifu Zhang, Huaping Liu, Fuchun Sun
备注：
has been accepted by Robotics: Science and Systems 2021
In this paper, we propose a novel task, Manipulation Question Answering (MQA), where the robot performs manipulation actions to change the environment in order to answer a given question. To solve this problem, a framework consisting of a QA module and a manipulation module is proposed. For the QA module, we adopt the method for the Visual Question Answering (VQA) task. For the manipulation module, a Deep Q Network (DQN) model is designed to generate manipulation actions for the robot to interact with the environment. We consider the situation where the robot continuously manipulating objects inside a bin until the answer to the question is found. Besides, a novel dataset that contains a variety of object models, scenarios and corresponding question-answer pairs is established in a simulation environment. Extensive experiments have been conducted to validate the effectiveness of the proposed framework.
Multi-hop Question Answering under Temporal Knowledge Editing
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；South China University of Technology（华南理工大学）；Ant Group（蚂蚁集团）；SDAIA-KAUST AI（沙特数据与人工智能局-阿卜杜拉国王科技大学人工智能项目）；Provable Responsible AI and Data Analytics (PRADA) Lab（可信赖负责任人工智能与数据分析实验室）
Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement
University of Oxford（牛津大学）；PyPOTS Research（PyPOTS研究）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Griffith University（格里菲斯大学）；Squirrel Ai Learning（Squirrel Ai学习）
Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answerin…
Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulation
The Future Network of Intelligence Institute, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）未来智能网络研究院）；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）
National University of Singapore（新加坡国立大学）；University of California, Santa Barbara（加州大学圣塔芭芭拉分校）
Obtaining training data for multi-hop question answering (QA) is time-consuming and resource-intensive. We explore the possibility to train a well-performed multi-hop QA model without referencing any human-labeled multi-hop question-answer pairs, i.e., unsupervised multi-hop QA. We propose MQA-QG, an unsupervised framework that can generate human-like multi-hop training data from both homogeneous and heterogeneous data sources. MQA-QG generates questions by first selecting/generating relevant information from each data source and then integrating the multiple information to form a multi-hop qu…
Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
Beihang University（北京航空航天大学）；Shanghai Innovation Institute（上海创新研究院）；Southern University of Science and Technology（南方科技大学）；Shanghai Jiao Tong University（上海交通大学）
3D Affordance Keypoint Detection for Robotic Manipulation
Advanced Robotics Centre, National University of Singapore（新加坡国立大学先进机器人中心）；Department of Mechanical Engineering, National University of Singapore（新加坡国立大学机械工程系）
RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
Xiaomi EV（小米电动车）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学 Gallagher人工智能学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）
Robotics at Google（谷歌机器人部门）；TU Berlin（柏林工业大学）；Google Research（谷歌研究院）
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, f…
ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Model
Midea Group（美的集团）；East China Normal University（华东师范大学）；Shanghai University（上海大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）；Tsinghua University（清华大学）
What Matters in Building Vision-Language-Action Models for Generalist Robots
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；ByteDance Research（字节跳动研究院）；CASIA MAIS-NLPR；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
TomTom（TomTom公司）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Saarland Informatics Campus（萨尔信息学园区）
Despite significant success in Visual Question Answering (VQA), VQA models have been shown to be notoriously brittle to linguistic variations in the questions. Due to deficiencies in models and datasets, today's models often rely on correlations rather than predictions that are causal w.r.t. data. In this paper, we propose a novel way to analyze and measure the robustness of the state of the art models w.r.t semantic visual variations as well as propose ways to make models more robust against spurious correlations. Our method performs automated semantic image manipulations and tests for consis…
AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation
School of Software Engineering, South China University of Technology（软件工程学院，华南理工大学）；School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
