Integrated Exploration and Sequential Manipulation on Scene Graph with LLM-based Situated Replanning
Beihang University（北航大学）；State Key Laboratory of General Artificial Intelligence（一般人工智能国家重点实验室）；University of California, Los Angeles（加州大学洛杉矶分校）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）
Heqing Yang, Ziyuan Jiao, Shu Wang, Yida Niu, Si Liu, Hangxin Liu
In partially known environments, robots must combine exploration to gather information with task planning for efficient execution. To address this challenge, we propose EPoG, an Exploration-based sequential manipulation Planning framework on Scene Graphs. EPoG integrates a graph-based global planner with a Large Language Model (LLM)-based situated local planner, continuously updating a belief graph using observations and LLM predictions to represent known and unknown objects. Action sequences are generated by computing graph edit operations between the goal and belief graphs, ordered by temporal dependencies and movement costs. This approach seamlessly combines exploration and sequential manipulation planning. In ablation studies across 46 realistic household scenes and 5 long-horizon daily object transportation tasks, EPoG achieved a success rate of 91.3%, reducing travel distance by 36.1% on average. Furthermore, a physical mobile manipulator successfully executed complex tasks in unknown and dynamic environments, demonstrating EPoG's potential for real-world applications.
Video Replanning via Latent Embedding Refinement and Rejection-Based Sampling
National Taiwan University（国立台湾大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Cornell University（康奈尔大学）；Harvard University（哈佛大学）；Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）
VLCP: Vision Language Control Policy Closed-Loop Code Replanning for Robot Manipulation
University of Monastir（莫纳斯提尔大学）；St. Mildred’s-Lightbourn School（圣米尔德雷德-莱特本学校）；Cornell University（康奈尔大学）；Carnegie Mellon University（卡内基梅隆大学）；Silverstream AI（银流人工智能公司）；Mila -- Quebec AI Institute（米拉-魁北克人工智能研究所）
Individualized manufacturing of cars requires kitting: the collection of individual sets of part variants for each car. This challenging logistic task is frequently performed manually by warehouseman. We propose a mobile manipulation robotic system for autonomous kitting, building on the Kuka Miiwa platform which consists of an omnidirectional base, a 7 DoF collaborative iiwa manipulator, cameras, and distance sensors. Software modules for detection and pose estimation of transport boxes, part segmentation in these containers, recognition of part variants, grasp generation, and arm trajectory …
CaFe-TeleVision: A Coarse-to-Fine Teleoperation System with Immersive Situated Visualization for Enhanced Ergonomics
Department of Mechanical and Automation Engineering, T-Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong SAR（机械与自动化工程系、T-Stone机器人研究所、香港中文大学（深圳））
AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；RoboScience（机器人科学）；East China Normal University（华东师范大学）；Peking University（北京大学）；Nanjing University（南京大学）
This paper develops closed-loop tactile controllers for dexterous robotic manipulation with a dual-palm robotic system. Tactile dexterity is an approach to dexterous manipulation that plans for robot/object interactions that render interpretable tactile information for control. We divide the role of tactile control into two goals: 1) control the contact state between the end-effector and the object (contact/no-contact, stick/slip) by regulating the stability of planned contact configurations and monitoring undesired slip events; and 2) control the object state by tactile-based tracking and ite…
Synergizing Efficiency and Reliability for Continuous Mobile Manipulation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
Commit While Futures Agree: Consequence-Aware Adaptive Action Chunking for Robot Manipulation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；ANU Intelligent（澳大利亚国立大学智能）；Victoria University of Wellington（惠灵顿维多利亚大学）；The Hong Kong University of Science and Technology（香港科技大学）；Southern University of Science and Technology（南方科技大学）
Carnegie Mellon University（卡内基梅隆大学）；Technion-Israel Institute of Technology（以色列理工学院）
In warehouse and manufacturing environments, manipulation platforms are frequently deployed at conveyor belts to perform pick and place tasks. Because objects on the conveyor belts are moving, robots have limited time to pick them up. This brings the requirement for fast and reliable motion planners that could provide provable real-time planning guarantees, which the existing algorithms do not provide. Besides the planning efficiency, the success of manipulation tasks relies heavily on the accuracy of the perception system which is often noisy, especially if the target objects are perceived fr…
HiTac-WAM: A Hierarchical Tactile World Action Model for Contact-Rich Robot Manipulation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ImprintX Robotics（ImprintX机器人公司）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Carnegie Mellon University（卡内基梅隆大学）；Technion-Israel Institute of Technology（以色列理工学院）
In warehouse and manufacturing environments, manipulation platforms are frequently deployed at conveyor belts to perform pick and place tasks. Because objects on the conveyor belts are moving, robots have limited time to pick them up. This brings the requirement for fast and reliable motion planners that could provide provable real-time planning guarantees, which the existing algorithms do not provide. Besides the planning efficiency, the success of manipulation tasks relies heavily on the accuracy of the perception system which is often noisy, especially if the target objects are perceived fr…
GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency
School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；University of Liverpool（利物浦大学）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算学系）；State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学海洋地球科学国家重点实验室，自动化与智能感知学院）
Modular Multi-Level Replanning TAMP Framework for Dynamic Environment
Research Center of the Satellite Technology, Harbin Institute of Technology（哈尔滨工业大学卫星技术研究中心）；Institute of Space Science and Applied Technology, Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳校区空间科学与应用技术研究院）
RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation
Institute of Trustworthy Embodied AI（可信具身人工智能研究院）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）；Singapore Management University（新加坡国立大学）
Korea Advanced Institute of Science and Technology（韩国科学技术院）
We present a task-and-motion planning (TAMP) algorithm robust against a human operator's cooperative or adversarial interventions. Interventions often invalidate the current plan and require replanning on the fly. Replanning can be computationally expensive and often interrupts seamless task execution. We introduce a dynamically reconfigurable planning methodology with behavior tree-based control strategies toward reactive TAMP, which takes the advantage of previous plans and incremental graph search during temporal logic-based reactive synthesis. Our algorithm also shows efficient recovery fu…
To solve multi-step manipulation tasks in the real world, an autonomous robot must take actions to observe its environment and react to unexpected observations. This may require opening a drawer to observe its contents or moving an object out of the way to examine the space behind it. Upon receiving a new observation, the robot must update its belief about the world and compute a new plan of action. In this work, we present an online planning and execution system for robots faced with these challenges. We perform deterministic cost-sensitive planning in the space of hybrid belief states to sel…
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）
With the advancement of robotics, machine learning, and machine perception, increasingly more robots will enter human environments to assist with daily tasks. However, dynamically-changing human environments requires reactive motion plans. Reactivity can be accomplished through replanning, e.g. model-predictive control, or through a reactive feedback policy that modifies on-going behavior in response to sensory events. In this paper, we investigate how to use machine learning to add reactivity to a previously learned nominal skilled behavior. We approach this by learning a reactive modificatio…
MPC-based Coarse-to-Fine Motion Planning for Robotic Object Transportation in Cluttered Environments
Department of Electrical and Computer Engineering, University of Kaiserslautern Landau（电气与计算机工程系，凯撒斯劳滕大学）；Department of Mechanical Engineering, Dalhousie University（机械工程系，达尔豪斯大学）
Look as You Leap: Planning Simultaneous Motion and Perception for High-DOF Robots
Department of Computer Science, Rice University（计算机科学系，里士大学）；Ken Kennedy Institute, Rice University（肯尼迪研究所，里士大学）；Houston Methodist Academic Institute（休斯顿 Methodist 学术研究院）
DAG-STL: A Hierarchical Framework for Zero-Shot Trajectory Planning under Signal Temporal Logic Specifications
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（自动化与智能感知学院，上海交通大学，上海，中国）；Institute of Artificial Intelligence, Xiamen University, Xiamen, China（人工智能研究院，厦门大学，厦门，中国）
