人类具备将手持物体作为工具与环境交互的卓越能力，实现这一能力的前提是人类能内在估算手部动作如何影响物体运动，我们希望赋予机器人该能力。本文提出一种方法，可通过外部相机拍摄的RGB图像，联合估算机器人抓取物体的几何形状与位姿。值得注意的是，该方法能将估算的几何形状转换至机器人坐标系，且无需校准外部相机的外参。\n我们的方法利用3D foundation models（3D基础模型，即针对3D视觉任务在海量数据集上预训练的大模型）生成手持物体的初始估算结果，这些初始估算不具备物理正确的尺度，且处于相机坐标系下。随后，我们构建并高效求解一个坐标对齐问题，以恢复准确的尺度，同时将物体转换至机器人坐标系。\n之后可基于机械臂关节角定义到物体上指定点的正向运动学映射，这些映射能估算任意构型下手持物体上的点，从而可根据被抓取物体上的坐标规划机器人运动。我们在一台抓取多种真实世界物体的机器人机械臂上，对所提方法进行了实证评估。
英文摘要：
Humans have the remarkable ability to use held objects as tools to interact with their environment. For this to occur, humans internally estimate how hand movements affect the object's movement. We wish to endow robots with this capability. We contribute methodology to jointly estimate the geometry and pose of objects grasped by a robot, from RGB images captured by an external camera. Notably, our method transforms the estimated geometry into the robot's coordinate frame, while not requiring the extrinsic parameters of the external camera to be calibrated. Our approach leverages 3D foundation models, large models pre-trained on huge datasets for 3D vision tasks, to produce initial estimates of the in-hand object. These initial estimations do not have physically correct scales and are in the camera's frame. Then, we formulate, and efficiently solve, a coordinate-alignment problem to recover accurate scales, along with a transformation of the objects to the coordinate frame of the robot. Forward kinematics mappings can subsequently be defined from the manipulator's joint angles to specified points on the object. These mappings enable the estimation of points on the held object at arbitrary configurations, enabling robot motion to be designed with respect to coordinates on the grasped objects. We empirically evaluate our approach on a robot manipulator holding a diverse set of real-world objects.
Multi-step manipulation task and motion planning guided by video demonstration
CIIRC, Czech Technical University in Prague（捷克技术大学普拉格分校智能信息研究中心）；LAAS-CNRS, Universite de Toulouse, CNRS, Toulouse（图卢兹大学CNRS实验室）；INRIA, Paris（巴黎国家信息与自动化研究所）
Joint estimation of grasped object pose and extrinsic contacts is central to robust and dexterous manipulation. In this paper, we propose a novel state-estimation algorithm that jointly estimates contact location and object pose in 3D using exclusively proprioception and tactile feedback. Our approach leverages two complementary particle filters: one to estimate contact location (CPFGrasp) and another to estimate object poses (SCOPE). We implement and evaluate our approach on real-world single-arm and dual-arm robotic systems. We demonstrate that by bringing two objects into contact, the robot…
MIT（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）；Magna International Inc.（麦格纳国际公司）
We propose a method that simultaneously estimates and controls extrinsic contact with tactile feedback. The method enables challenging manipulation tasks that require controlling light forces and accurate motions in contact, such as balancing an unknown object on a thin rod standing upright. A factor graph-based framework fuses a sequence of tactile and kinematic measurements to estimate and control the interaction between gripper-object-environment, including the location and wrench at the extrinsic contact between the grasped object and the environment and the grasp wrench transferred from t…
Prioritized Planning for Target-Oriented Manipulation via Hierarchical Stacking Relationship Prediction
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Application（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究所）；Xi’an Jiaotong University（西安交通大学）
University of Birmingham（伯明翰大学）；Chalmers University of Technology（查尔姆斯理工大学）；University College London（伦敦大学学院）
This paper addresses the problem of simultaneously exploring an unknown object to model its shape, using tactile sensors on robotic fingers, while also improving finger placement to optimise grasp stability. In many situations, a robot will have only a partial camera view of the near side of an observed object, for which the far side remains occluded. We show how an initial grasp attempt, based on an initial guess of the overall object shape, yields tactile glances of the far side of the object which enable the shape estimate and consequently the successive grasps to be improved. We propose a …
Robotics Institute（机器人研究所）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Electrical Engineering and Computer Science（电子工程与计算机科学学院）
To be useful in everyday environments, robots must be able to observe and learn about objects. Recent datasets enable progress for classifying data into known object categories; however, it is unclear how to collect reliable object data when operating in cluttered, partially-observable environments. In this paper, we address the problem of building complete 3D models for real-world objects using a robot platform, which can remove objects from clutter for better classification. Furthermore, we are able to learn entirely new object categories as they are encountered, enabling the robot to classi…
A Vision-Based Collision Sensing Method for Stable Circular Object Grasping with A Soft Gripper System
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（电子与计算机工程系，香港科学与技术大学）；Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（土木与环境工程系，香港科学与技术大学）
Lan-grasp: Using Large Language Models for Semantic Object Grasping and Placement
Department of Computer Science and Artificial Intelligence, University of Technology Nuremberg（计算机科学与人工智能系，图恩技术大学）；Department of Engineering ``Enzo Ferrari'' (DIEF), University of Modena and Reggio Emilia（恩佐·费尔拉蒂工程系（DIEF），摩德纳和雷吉奥艾米利亚大学）；Technical University of Munich（慕尼黑技术大学）
3D Foundation Model-Based Loop Closing for Decentralized Collaborative SLAM
Department of Computer and Software Engineering, Polytechnique Montréal（计算机与软件工程系，蒙特利尔理工学院）；Oxford Robotics Institute, Dept. of Engineering Science, University of Oxford（牛津大学机器人研究所，工程科学系）
Simultaneous Time Synchronization and Mutual Localization for Multi-robot System
Polytechnic Institute of Zhejiang University（浙江大学理工学院）；Huzhou Institute of Zhejiang University（浙江大学湖州研究院）；Institute of Cyber-Systems and Control, Zhejiang University（浙江大学控制科学与工程学院）；The Hong Kong Polytechnic University（香港理工大学）；Collegue of Optical Science and Engineering, Zhejiang University（浙江大学光电科学与工程学院）
Instituto de Investigación en Ingeniería de Aragón (I3A)（阿拉贡工程研究院（I3A））；Universidad de Zaragoza（萨拉戈萨大学）；Massachusetts Institute of Technology（麻省理工学院）；Genius Sports（Genius体育）；UC San Diego（加州大学圣地亚哥分校）；Technion – Israel Institute of Technology（以色列理工学院）
Active Simultaneous Localization and Mapping (SLAM) is the problem of planning and controlling the motion of a robot to build the most accurate and complete model of the surrounding environment. Since the first foundational work in active perception appeared, more than three decades ago, this field has received increasing attention across different scientific communities. This has brought about many different approaches and formulations, and makes a review of the current trends necessary and extremely valuable for both new and experienced researchers. In this work, we survey the state-of-the-a…
