We consider the task of generating segmentation masks for the target object from an object manipulation instruction, which allows users to give open vocabulary instructions to domestic service robots. Conventional segmentation generation approaches often fail to account for objects outside the camera's field of view and cases in which the order of vertices differs but still represents the same polygon, which leads to erroneous mask generation. In this study, we propose a novel method that generates segmentation masks from open vocabulary instructions. We implement a novel loss function using optimal transport to prevent significant loss where the order of vertices differs but still represents the same polygon. To evaluate our approach, we constructed a new dataset based on the REVERIE dataset and Matterport3D dataset. The results demonstrated the effectiveness of the proposed method compared with existing mask generation methods. Remarkably, our best model achieved a +16.32% improvement on the dataset compared with a representative polygon-based method.
AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；BIGAI（北京智源人工智能研究院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation
AIML, Adelaide University（AIML，阿德莱德大学）；East China Normal University（华东师范大学）；NERC-RVC, Hunan University（NERC-RVC，湖南大学）；The University of Western Australia（西澳大学）；Breaker Industries
OpenMap: Instruction Grounding via Open-Vocabulary Visual-Language Mapping
School of Software, Tsinghua University（清华大学软件学院）；School of computer science and engineering, Central South University（中南大学计算机科学与工程学院）；Inspur Yunzhou Industrial Internet Co., Ltd（Inspur Yunzhou工业互联网有限公司）；Tsinghua University（清华大学）
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable sc…
In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Department of Networked Intelligence, Peng Cheng Laboratory（鹏城实验室网络智能部）；School of Computer Science and Engineering, Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）计算机科学与工程学院）
GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping
Beihang University（北京航空航天大学）；EncoSmart（恩科智能）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；AIR, Tsinghua University（清华大学AIR）；Imperial College London（帝国理工学院）；China Mobile Research Institute（中国移动研究院）；Wuhan University（武汉大学）；Shanghai AI Laboratory（上海人工智能实验室）；the University of Hong Kong（香港大学）
AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Peking University（北京大学）
A Multimodal Depth-Aware Method For Embodied Reference Understanding
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；KIT Campus Transfer GmbH (KCT)（KIT校园转移有限公司）；Istanbul Technical University（伊斯坦布尔技术大学）；Carnegie Mellon University（卡内基梅隆大学）
SuZhou Automotive Research Institute of Tsinghua University（清华大学苏州汽车研究院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Hyundai Motor Advanced Tech. R&D Center；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）
R2F: Repurposing Ray Frontiers for LLM-free Object Navigation
Department of Computer, Automation and Management Engineering, Sapienza University of Rome（罗马萨皮恩扎大学计算机、自动化与管理工程系）；Department of Electronics and Automation, Mohamed Khider University of Biskra（比斯克拉Mohamed Khider大学电子与自动化系）；International University of Rome UNINT, Rome（罗马国际大学UNINT）
