Anytime a robot manipulator is controlled via visual feedback, the transformation between the robot and camera frame must be known. However, in the case where cameras can only capture a portion of the robot manipulator in order to better perceive the environment being interacted with, there is greater sensitivity to errors in calibration of the base-to-camera transform. A secondary source of uncertainty during robotic control are inaccuracies in joint angle measurements which can be caused by biases in positioning and complex transmission effects such as backlash and cable stretch. In this work, we bring together these two sets of unknown parameters into a unified problem formulation when the kinematic chain is partially visible in the camera view. We prove that these parameters are non-identifiable implying that explicit estimation of them is infeasible. To overcome this, we derive a smaller set of parameters we call Lumped Error since it lumps together the errors of calibration and joint angle measurements. A particle filter method is presented and tested in simulation and on two real world robots to estimate the Lumped Error and show the efficiency of this parameter reduction.
The Hong Kong University of Science and Technology（香港科技大学）
Visible light positioning (VLP) technology is a promising technique as it can provide high accuracy positioning based on the existing lighting infrastructure. However, existing approaches often require dense lighting distributions. Additionally, due to complicated indoor environments, it is still challenging to develop a robust VLP. In this work, we proposed loosely-coupled multi-sensor fusion method based on VLP and Simultaneous Localization and Mapping (SLAM), with light detection and ranging (LiDAR), odometry, and rolling shutter camera. Our method can provide accurate and robust robotics l…
University of Southern California（南加利福尼亚大学）；HomeAway, Inc.（HomeAway公司）；Indiana University - Bloomington（印第安纳大学伯明顿分校）
We investigate the scenario that a robot needs to reach a designated goal after taking a sequence of appropriate actions in a non-static environment that is partially structured. One application example is to control a marine vehicle to move in the ocean. The ocean environment is dynamic and oftentimes the ocean waves result in strong disturbances that can disturb the vehicle's motion. Modeling such dynamic environment is non-trivial, and integrating such model in the robotic motion control is particularly difficult. Fortunately, the ocean currents usually form some local patterns (e.g. vortex…
Zhejiang University（浙江大学）；Huzhou Institute of Zhejiang University（浙江大学湖州研究院）
Mutual localization provides a consensus of reference frame as an essential basis for cooperation in multirobot systems. Previous works have developed certifiable and robust solvers for relative transformation estimation between each pair of robots. However, recovering relative poses for robotic swarm with partially mutual observations is still an unexploited problem. In this paper, we present a complete algorithm for it with optimality, scalability and robustness. Firstly, we fuse all odometry and bearing measurements in a unified minimization problem among the Stiefel manifold. Furthermore, …
A neural network based flexible object manipulation system for a humanoid robot on FPGA is proposed. Although the manipulations of flexible objects using robots attract ever increasing attention since these tasks are the basic and essential activities in our daily life, it has been put into practice only recently with the help of deep neural networks. However such systems have relied on GPU accelerators, which cannot be implemented into the space limited robotic body. Although field programmable gate arrays (FPGAs) are known to be energy efficient and suitable for embedded systems, the model s…
Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics
Department of Automation, University of Science and Technology of China（自动化系，中国科学技术大学）；SmartMore Corporation（SmartMore公司）；Cyberspace Institute of Advanced Technology, Guangzhou University（广西亚技术空间研究所，广州大学）；th Medical Center of Chinese PLA General Hospital（中国人民解放军总医院第八医学中心）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception
State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology（数字制造装备与技术国家重点实验室，机械科学与工程学院，华中科技大学）
Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation
University of Science and Technology of China（中国科学技术大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Shanghai Jiaotong University（上海交通大学）；Nanjing University（南京大学）
Massachusetts Institute of Technology（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Representing and reasoning about uncertainty is crucial for autonomous agents acting in partially observable environments with noisy sensors. Partially observable Markov decision processes (POMDPs) serve as a general framework for representing problems in which uncertainty is an important factor. Online sample-based POMDP methods have emerged as efficient approaches to solving large POMDPs and have been shown to extend to continuous domains. However, these solutions struggle to find long-horizon plans in problems with significant uncertainty. Exploration heuristics can help guide planning, but…
Massachusetts Institute of Technology（麻省理工学院）；Woods Hole Oceanographic Institution（伍兹霍尔海洋研究所）
We present PLUMES, a planner to localizing and collecting samples at the global maximum of an a priori unknown and partially observable continuous environment. The "maximum-seek-and-sample" (MSS) problem is pervasive in the environmental and earth sciences. Experts want to collect scientifically valuable samples at an environmental maximum (e.g., an oil-spill source), but do not have prior knowledge about the phenomenon's distribution. We formulate the MSS problem as a partially-observable Markov decision process (POMDP) with continuous state and observation spaces, and a sparse reward signal.…
National Research Council（国家研究委员会）；University of Salento（萨伦托大学）；Danish Technological Institute（丹麦技术研究所）
Accurate soil mapping is critical for a highly-automated agricultural vehicle to successfully accomplish important tasks including seeding, ploughing, fertilising and controlled traffic, with limited human supervision, ensuring at the same time high safety standards. In this research, a multi-sensor ground mapping and characterisation approach is proposed, whereby data coming from heterogeneous but complementary sensors, mounted on-board an unmanned rover, are combined to generate a multi-layer map of the environment and specifically of the supporting ground. The sensor suite comprises both ex…
LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusion
school of Data Science, Fudan University（复旦大学数据科学学院）；Fudan University（复旦大学）；School of statistics and information, Shanghai University of International Business and Economics（上海对外经贸大学统计与信息学院）
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
In this work, a control scheme for approaching and unveiling a partially occluded object of interest is proposed.The control scheme is based only on the classified point cloud obtained by the in-hand camera attached to the robot's end effector. It is shown that the proposed controller reaches in the vicinity of the object progressively unveiling the neighborhood of each visible point of the object of interest. It can therefore potentially achieve the complete unveiling of the object. The proposed control scheme is evaluated through simulations and experiments with a UR5e robot with an in-hand …
Universität Hamburg（汉堡大学）；National University of Singapore（新加坡国立大学）；Aalto University（阿尔托大学）；Technische Universität Darmstadt（达姆施塔特工业大学）
Noisy sensing, imperfect control, and environment changes are defining characteristics of many real-world robot tasks. The partially observable Markov decision process (POMDP) provides a principled mathematical framework for modeling and solving robot decision and control tasks under uncertainty. Over the last decade, it has seen many successful applications, spanning localization and navigation, search and tracking, autonomous driving, multi-robot systems, manipulation, and human-robot interaction. This survey aims to bridge the gap between the development of POMDP models and algorithms at on…
Robotic manipulation of cloth remains challenging for robotics due to the complex dynamics of the cloth, lack of a low-dimensional state representation, and self-occlusions. In contrast to previous model-based approaches that learn a pixel-based dynamics model or a compressed latent vector dynamics, we propose to learn a particle-based dynamics model from a partial point cloud observation. To overcome the challenges of partial observability, we infer which visible points are connected on the underlying cloth mesh. We then learn a dynamics model over this visible connectivity graph. Compared to…
University of Twente（特文特大学）；University of Genoa（热那亚大学）；Istituto Italiano di Tecnologia（意大利技术研究院）；Politecnico di Milano（米兰理工大学）
A biopsy is the only diagnostic procedure for accurate histological confirmation of breast cancer. When sonographic placement is not feasible, a Magnetic Resonance Imaging(MRI)-guided biopsy is often preferred. The lack of real-time imaging information and the deformations of the breast make it challenging to bring the needle precisely towards the tumour detected in pre-interventional Magnetic Resonance (MR) images. The current manual MRI-guided biopsy workflow is inaccurate and would benefit from a technique that allows real-time tracking and localisation of the tumour lesion during needle in…
Learning object manipulation is a critical skill for robots to interact with their environment. Even though there has been significant progress in robotic manipulation of rigid objects, interacting with non-rigid objects remains challenging for robots. In this work, we introduce velcro peeling as a representative application for robotic manipulation of non-rigid objects in complex environments. We present a method of learning force-based manipulation from noisy and incomplete sensor inputs in partially observable environments by modeling long term dependencies between measurements with a multi…
University of Bourgogne Franche-Comté（勃艮第-弗朗什-孔泰大学）；FEMTO-ST Institute（FEMTO-ST研究所）；CNRS（法国国家科学研究中心）；Warsaw University of Technology（华沙理工大学）；Institute of Fundamental Technological Research Polish Academy of Sciences（波兰科学院基础技术研究所）；University of Luxembourg（卢森堡大学）
We present a distributed framework for predicting whether a planned reconfiguration step of a modular robot will mechanically overload the structure, causing it to break or lose stability under its own weight. The algorithm is executed by the modular robot itself and based on a distributed iterative solution of mechanical equilibrium equations derived from a simplified model of the robot. The model treats inter-modular connections as beams and assumes no-sliding contact between the modules and the ground. We also provide a procedure for simplified instability detection. The algorithm is verifi…
We present joint learning of instance and semantic segmentation for visible and occluded region masks. Sharing the feature extractor with instance occlusion segmentation, we introduce semantic occlusion segmentation into the instance segmentation model. This joint learning fuses the instance- and image-level reasoning of the mask prediction on the different segmentation tasks, which was missing in the previous work of learning instance segmentation only (instance-only). In the experiments, we evaluated the proposed joint learning comparing the instance-only learning on the test dataset. We als…
Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via dialogues. Our approach first predicts target scores and attribute scores through vision-and-language grounding. To handle ambiguous objects and commands, we propose an attribute-guided formulation of t…
Cartesian impedance control is a type of motion control strategy for robots that improves safety in partially unknown environments by achieving a compliant behavior of the robot with respect to its external forces. This compliant robot behavior has the added benefit of allowing physical human guidance of the robot. In this paper, we propose a C++ implementation of compliance control valid for any torque-commanded robotic manipulator. The proposed controller implements Cartesian impedance control to track a desired end-effector pose. Additionally, joint impedance is projected in the nullspace o…
Look-to-Touch: A Vision-Enhanced Proximity and Tactile Sensor for Distance and Geometry Perception in Robotic Manipulation
State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University（上海交通大学机械系统与振动国家重点实验室）；Robotics Institute, School of Mechanical Engineering, Shanghai Jiao Tong University（上海交通大学机械工程学院机器人研究所）
Consider the goal of visiting every part of a room that is not blocked by obstacles. Doing so efficiently requires both sensors and planning. Our findings suggest a method of inexpensive optical range finding for robotic room traversal. Our room traversal algorithm relies upon the approximate distance from the robot to the nearest obstacle in 360 degrees. We then choose the path with the furthest approximate distance. Since millimeter-precision is not required for our problem, we have opted to develop our own laser range finding solution, in lieu of using more common, but also expensive soluti…
Optimal Robotic Velcro Peeling with Force Feedback
Department of Electrical and Computer Engineering, University of Minnesota（电气与计算机工程系，明尼苏达大学）；Department of Computer Science and Engineering, University of Minnesota（计算机科学与工程系，明尼苏达大学）
University of Tennessee（田纳西大学）；Miami University（迈阿密大学）；University of Tennessee Health Science Center（田纳西大学健康科学中心）；University of Iowa（爱荷华大学）
People with Alzheimer's disease and related dementias (ADRD) often show the problem of repetitive questioning, which brings a great burden on persons with ADRD (PwDs) and their caregivers. Conversational robots hold promise of coping with this problem and hence alleviating the burdens on caregivers. In this paper, we proposed a partially observable markov decision process (POMDP) model for the PwD-robot interaction in the context of repetitive questioning, and used Q-learning to learn an adaptive conversation strategy (i.e., rate of follow-up question and difficulty of follow-up question) towa…
Unveiling the Potential of iMarkers: Invisible Fiducial Markers for Advanced Robotics
Automation and Robotics Research Group, Interdisciplinary Centre for Security, Reliability, and Trust (SnT), University of Luxembourg（自动化与机器人研究组，安全、可靠性与信任跨学科中心（SnT），卢森堡大学）；Faculty of Science, Technology, and Medicine, University of Luxembourg（科学、技术与医学学院，卢森堡大学）；Department of Physics & Materials Science, University of Luxembourg（物理与材料科学系，卢森堡大学）；Institute for Advanced Studies, University of Luxembourg（先进研究学院，卢森堡大学）
Technical University of Munich（慕尼黑工业大学）；Toyota Motor Europe（丰田汽车欧洲公司）
The ability to successfully grasp objects is crucial in robotics, as it enables several interactive downstream applications. To this end, most approaches either compute the full 6D pose for the object of interest or learn to predict a set of grasping points. While the former approaches do not scale well to multiple object instances or classes yet, the latter require large annotated datasets and are hampered by their poor generalization capabilities to new geometries. To overcome these shortcomings, we propose to teach a robot how to grasp an object with a simple and short human demonstration. …
Georgia Institute of Technology（佐治亚理工学院）；Stanford University（斯坦福大学）
Simulation provides a safe and efficient way to generate useful data for learning complex robotic tasks. However, matching simulation and real-world dynamics can be quite challenging, especially for systems that have a large number of unobserved or unmeasurable parameters, which may lie in the robot dynamics itself or in the environment with which the robot interacts. We introduce a novel approach to tackle such a sim-to-real problem by developing policies capable of adapting to new environments, in a zero-shot manner. Key to our approach is an error-aware policy (EAP) that is explicitly made …
With an ever-widening domain of aerial robotic applications, including many mission critical tasks such as disaster response operations, search and rescue missions and infrastructure inspections taking place in GPS-denied environments, the need for reliable autonomous operation of aerial robots has become crucial. Operating in GPS-denied areas aerial robots rely on a multitude of sensors to localize and navigate. Visible spectrum cameras are the most commonly used sensors due to their low cost and weight. However, in environments that are visually-degraded such as in conditions of poor illumin…
In this paper, we propose a leader-follower hierarchical strategy for two robots collaboratively transporting an object in a partially known environment with obstacles. Both robots sense the local surrounding environment and react to obstacles in their proximity. We consider no explicit communication, so the local environment information and the control actions are not shared between the robots. At any given time step, the leader solves a model predictive control (MPC) problem with its known set of obstacles and plans a feasible trajectory to complete the task. The follower estimates the input…
