Open-vocabulary panoptic reconstruction is crucial for advanced robotics and simulation. However, existing 3D reconstruction methods, such as NeRF or Gaussian Splatting variants, often struggle to achieve the real-time inference frequency required by robotic control loops. Existing methods incur prohibitive latency when processing the high-dimensional features required for robust open-vocabulary segmentation. We propose Fast-SegSim, a novel, simple, and end-to-end framework built upon 2D Gaussian Splatting, designed to realize real-time, high-fidelity, and 3D-consistent open-vocabulary segmentation reconstruction. Our core contribution is a highly optimized rendering pipeline that specifically addresses the computational bottleneck of high-channel segmentation feature accumulation. We introduce two key optimizations: Precise Tile Intersection to reduce rasterization redundancy, and a novel Top-K Hard Selection strategy. This strategy leverages the geometric sparsity inherent in the 2D Gaussian representation to greatly simplify feature accumulation and alleviate bandwidth limitations, achieving render rates exceeding 40 FPS. Fast-SegSim provides critical value in robotic applications: it serves both as a high-frequency sensor input for simulation platforms like Gazebo, and its 3D-consistent outputs provide essential multi-view 'ground truth' labels for fine-tuning downstream perception tasks. We demonstrate this utility by using the generated labels to fine-tune the perception module in object goal navigation, successfully doubling the navigation success rate. Our superior rendering speed and practical utility underscore Fast-SegSim's potential to bridge the sim-to-real gap.
Depth-Only Open-Vocabulary 3D Semantic Segmentation For Privacy-Preserving Robotic Applications
Humanoid Robots Lab, University of Bonn(波恩大学仿人机器人实验室);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习和人工智能研究所);Center for Robotics, Bonn, Germany(波恩机器人中心)
Language-Conditioned Affordance-Pose Detection in 3D Point Clouds
FPT Software AI Center(FPT软件人工智能中心);TU Wien(维也纳工业大学);Imperial College London(帝国理工学院);University of Arkansas(阿肯色大学);Ton Duc Thang University(孙德胜大学);University of Science, Vietnam(越南科学大学);University of Liverpool(利物浦大学)
Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks
Department of Applied Informatics, Faculty of Mathematics, Physics and Informatics, Comenius University(应用信息学院,数学物理学院,科门ius大学);Knowledge Technology Group, Department of Informatics, University of Hamburg(知识技术组,信息学院,汉堡大学)
QueryCAD: Grounded Question Answering for CAD Models
ArtiMinds Robotics;IAS Lab, Computer Science Department, TU Darmstadt(IAS实验室,计算机科学系,图腾斯泰特大学);AICOR Institute for Artificial Intelligence, University of Bremen(AICOR人工智能研究所,不莱梅大学);Stuttgart University of Applied Sciences(斯图加特应用科学大学)
OTAS: Open-vocabulary Token Alignment for Outdoor Segmentation
Graz University of Technology, Faculty of Computer Science and Biomedical Engineering, Institute of Software Engineering and Artificial Intelligence(格拉茨技术大学,计算机科学与生物医学工程学院,软件工程与人工智能研究所);University of Applied Sciences Technikum Wien, Faculty of Industrial Engineering, Research Group Digital Manufacturing, Automation and Robotics(应用科学大学技术学院,工业工程学院,数字制造、自动化与机器人研究组);University of Natural Resources and Life Sciences, Department of Integrative Biology and Biodiversity Research, Institute for Integrative Nature Conservation Research(自然资源与生命科学大学,整合生物学与生物多样性研究部门,整合自然保护研究 institute)
OGScene3D: Incremental Open-Vocabulary 3D Gaussian Scene Graph Mapping for Scene Understanding
Shanghai Jiao Tong University(上海交通大学);University of Cambridge(剑桥大学);University of Technology Nuremberg(纽伦堡技术大学);Nanyang Technological University(南洋理工大学);Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University(自动化系,教育部系统控制与信息处理重点实验室,航空系统集成与航空系统-of-系统综合国家重点实验室,上海导航与定位基于服务重点实验室,上海交通大学)
GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping
Beihang University(北京航空航天大学);EncoSmart(恩科智能);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);AIR, Tsinghua University(清华大学AIR);Imperial College London(帝国理工学院);China Mobile Research Institute(中国移动研究院);Wuhan University(武汉大学);Shanghai AI Laboratory(上海人工智能实验室);the University of Hong Kong(香港大学)
OpenTie: Open-vocabulary Sequential Rebar Tying System
Hong Kong Center for Construction Robotics, The Hong Kong University of Science and Technology(香港建设机器人中心,香港科技大学);Division of Business and Hospitality Management (BHM), College of Professional and Continuing Education (CPCE), The Hong Kong Polytechnic University (PolyU)(专业及持续教育学院(CPCE)商务及酒店管理系,香港理工大学)
From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation
University of São Paulo(圣保罗大学);Instituto Israelita de Ensino e Pesquisa Albert Einstein(以色列爱因斯坦教育与研究机构);Graduate School of Information Sciences, Tohoku University(东北大学信息科学研究生院)
Open-Vocabulary Affordance Detection in 3D Point Clouds
FPT Software AI Center(FPT软件人工智能中心);VNUHCM-University of Science(越南胡志明市国家大学-科学大学);Automation & Control Institute (ACIN), TU Wien(维也纳工业大学自动化与控制研究所);Center for Vision, Automation & Control, AIT Austrian Institute of Technology (GmbH)(奥地利技术研究院(AIT)视觉、自动化与控制中心);Ton Duc Thang University(孙德胜大学);University of Arkansas(阿肯色大学);University of Liverpool(利物浦大学)
Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
Autonomous Intelligent Systems, University of Bonn(博恩大学自主智能系统中心);Lamarr Institute for Machine Learning and AI(拉马尔人工智能与机器学习研究所);Center for Robotics, University of Bonn(博恩大学机器人中心)
CLASP: Closed-loop Asynchronous Spatial Perception for Open-vocabulary Desktop Object Grasping
Harbin Institute of Technology(哈尔滨工业大学);National Key Laboratory of Smart Farm Technologies and Systems(智慧农场技术与系统全国重点实验室);Peng Cheng Laboratory(鹏城实验室);Northeastern University(东北大学)
Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge
Georgia Tech(佐治亚理工学院);Hello Robot Inc(Hello Robot公司);Carnegie Mellon(卡内基梅隆大学);Bielefeld University(比勒费尔德大学);Ukrainian Catholic University(乌克兰天主教大学);Bosch Center for AI(博世AI中心);IIITA(印度信息技术与管理大学阿拉哈巴德分校);IIT ISM Dhanbad(印度理工学院达纳巴德矿业与技术学院);IIITM(印度信息技术与管理大学古瓦哈蒂分校);CNAEIT(国家先进信息技术研究中心)
FSUNav: A Cerebrum-Cerebellum Architecture for Fast, Safe, and Universal Zero-Shot Goal-Oriented Navigation
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);College of Information Science and Technology, Eastern Institute of Technology(东方理工学院信息科学与技术学院)
Learning Fine-Grained Correspondence with Cross-Perspective Perception for Open-Vocabulary 6D Object Pose Estimation
School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University(人工智能与机器人学院和机器人视觉感知与控制技术国家工程研究中心,湖南大学);State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University(自主智能无人系统国家重点实验室,同济大学);School of Computer Science and Engineering, Hunan University of Science and Technology(计算机科学与工程学院,湖南科技大学)
R2F: Repurposing Ray Frontiers for LLM-free Object Navigation
Department of Computer, Automation and Management Engineering, Sapienza University of Rome(罗马萨皮恩扎大学计算机、自动化与管理工程系);Department of Electronics and Automation, Mohamed Khider University of Biskra(比斯克拉Mohamed Khider大学电子与自动化系);International University of Rome UNINT, Rome(罗马国际大学UNINT)