Visual prediction models are a promising solution for visual-based robotic grasping of cluttered, unknown soft objects. Previous models from the literature are computationally greedy, which limits reproducibility; although some consider stochasticity in the prediction model, it is often too weak to catch the reality of robotics experiments involving grasping such objects. Furthermore, previous work focused on elementary movements that are not efficient to reason in terms of more complex semantic actions. To address these limitations, we propose VP-GO, a ``light'' stochastic action-conditioned visual prediction model. We propose a hierarchical decomposition of semantic grasping and manipulation actions into elementary end-effector movements, to ensure compatibility with existing models and datasets for visual prediction of robotic actions such as RoboNet. We also record and release a new open dataset for visual prediction of object grasping, called PandaGrasp. Our model can be pre-trained on RoboNet and fine-tuned on PandaGrasp, and performs similarly to more complex models in terms of signal prediction metrics. Qualitatively, it outperforms when predicting the outcome of complex grasps performed by our robot.
DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning
Imprintx Robotics(印迹机器人);Electric Power Research Institute of State Grid Jibei Electric Power Co., Ltd.(国网冀北电力有限公司电力科学研究院);North China Electric Power Research Institute Co., Ltd.(华北电力科学研究院有限责任公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation
Peking University(北京大学);National University of Singapore(新加坡国立大学);Columbia University(哥伦比亚大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon(亚马逊公司)
One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)理工学院);Shenzhen Research Institute of Big Data (SRIBD)(深圳大数据研究院)
该研究推出真实机器人基准One Word, Different Action,评估具身系统的决策不变性与敏感性,发现现代模型在多任务约束整合为机器人动作时存在明显退化,核心挑战是组合多任务需求为正确决策。
Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
Technical University of Munich(慕尼黑技术大学);Corporate Research, Robert Bosch GmbH(罗伯特·博世集团企业研究部);University of California Berkeley(加州大学伯克利分校);Microsoft(微软);Google DeepMind(谷歌DeepMind);Carnegie Mellon University(卡内基梅隆大学);Imperial College London(伦敦帝国理工学院);Princeton University(普林斯顿大学);University of Southern California(南加州大学);Sun Yat-sen University(中山大学);University of Michigan(密歇根大学);Institute for Artificial Intelligence, University of Stuttgart(斯图加特大学人工智能研究所);The State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室)
NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation
The Hong Kong University of Science and Technology(香港科学与技术大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));National University of Singapore(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学)
WorldPlanner: Monte Carlo Tree Search and MPC with Action-Conditioned Visual World Models
Center for Robotics and Embodied Intelligence, Tandon School of Engineering, New York University, Brooklyn, NY(机器人与具身智能中心,工程学院,纽约大学,布鲁克林,纽约);FAIR at Meta(Meta的FAIR)
Invisible Servoing: a Visual Servoing Approach with Return-Conditioned Latent Diffusion
Robotics and Mechatronics group, Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente(特文特大学机器人与机电组,电气工程、数学与计算机科学学院);Department of Applied Mathematics, University of Twente(特文特大学应用数学系);Department of Computer, Control and Management Engineering, Sapienza University of Rome(罗马萨皮恩扎大学计算机、控制与管理工程系)
Action-Conditioned World Model for Goal Plane Guidance in Robotic Neck Ultrasound
Department of Mechanical and Automation Engineering, Chinese University of Hong Kong(香港中文大学机械与自动化工程学系);Department of Biomedical Engineering, City University of Hong Kong(香港城市大学生物医学工程学系);Centre for Artificial Intelligence and Robotics Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Surgical and Interventional Engineering, King’s College London(伦敦国王大学外科与介入工程学系);Department of Vascular Ultrasound, Xuanwu Hospital, the Capital Medical University(首都医科大学宣武医院血管超声科)
ERRA: An Embodied Representation and Reasoning Architecture for Long-horizon Language-conditioned Manipulation Tasks
The Hong Kong University of Science and Technology(香港科技大学);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(香港科技大学深港协同创新研究院);Monash University(莫纳什大学)
J-LAW: Joint Localization and Action-Conditioned World Modeling via Coupled Latent Factor Graphs
Geely Technology Europe(吉利技术欧洲公司);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)(信息工程测绘与遥感国家重点实验室(LIESMARS));Wuhan University(武汉大学)
Action- and Language-Conditioned Video Assessment for Embodied Control
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院电气工程学院)
Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning
Peking University(北京大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);New York University(纽约大学);University of Electronic Science and Technology of China(电子科技大学);Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学)