Symbols representing abstract states such as "dish in dishwasher" or "cup on table" allow robots to reason over long horizons by hiding details unnecessary for high-level planning. Current methods for learning to identify symbolic states in visual data require large amounts of labeled training data, but manually annotating such datasets is prohibitively expensive due to the combinatorial number of predicates in images. We propose a novel method for automatically labeling symbolic states in large-scale video activity datasets by exploiting known pre- and post-conditions of actions. This automatic labeling scheme only requires weak supervision in the form of an action label that describes which action is demonstrated in each video. We use our framework to train predicate classifiers to identify symbolic relationships between objects when prompted with object bounding boxes, and demonstrate that such predicate classifiers can match the performance of those trained with full supervision at a fraction of the labeling cost. We also apply our framework to an existing large-scale human activity dataset, and demonstrate the ability of these predicate classifiers trained on human data to enable closed-loop task planning in the real world.
Unifying Deep Predicate Invention with Pre-trained Foundation Models
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Computer Science and Engineering Division, University of Michigan（密歇根大学计算机科学与工程系）；Department of Computer Science, University of Pittsburgh（匹兹堡大学计算机科学系）；Centaur AI Institute（Centaur人工智能研究所）；Department of Electrical and Computer Engineering, Princeton University（普林斯顿大学电气与计算机工程系）
Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding
University of Arkansas（阿拉巴马大学）；National University of Singapore（新加坡国立大学）；TU Wien（维也纳技术大学）；Max Planck Research School for Intelligent Systems（智能系统马克斯·普朗克研究学校）；University of Stuttgart（斯图加特大学）；University of Liverpool（利物浦大学）
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Lenovo CTO Organization（联想首席技术官组织）；Nanyang Technological University（南洋理工大学）；TeleAl, China Telecom（中国电信天翼人工智能公司）
Reinforced Grounded Action Transformation for Sim-to-Real Transfer
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Computer Sciences Department, University of Wisconsin—Madison（威斯康星大学麦迪逊分校计算机科学系）；Army Research Laboratory（美国陆军研究实验室）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sony AI（索尼人工智能）
HaReCAP: Habitual-action Grounding for Recursive Large Language Model Agents
North China Institute of Computer System Engineering（华北计算机系统工程研究所）；University of Science and Technology of China（中国科学技术大学）；China Information Security Research Institute Co., Ltd.（中国信息安全研究院有限公司）
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Midea Group（美的集团）；The Hong Kong University of Science and Technology（香港科技大学）
Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policy
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Shanghai AI Lab（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Nanjing University（南京大学）；Tsinghua University（清华大学）
Grounding Foundational Vision Models with 3D Human Poses for Robust Action Recognition
Arizona State University（亚利桑那州立大学）；Emory University（埃默里大学）；University of California, Berkeley（加州大学伯克利分校）；Delft University of Technology（代尔夫特理工大学）；Algoverse AI Research（Algoverse AI 研究）
Grounded in Time: A Multi-Source Dataset and Benchmark for Temporal Grounding in Robotic Manipulation
South China University of Technology（华南理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Northwestern Polytechnical University（西北工业大学）；The Hong Kong Polytechnic University（香港理工大学）
