Data-driven approaches for modelling contact-rich tasks address many of the difficulties that analytical models bear. For real-world scenarios, the hardware capabilities constrain the available measurements and consequently, every step of the problem's formulation. In this work, we propose a formulation that encapsulates knowledge from a baseline controller for the contact-rich task of food-cutting. Based on this formulation, we employ deep networks to model the dynamics within a model predictive controller. We design a training process based on curriculum training with learning rate decay for multi-step predictions, which are essential for receding horizon control. Experimental results demonstrate that even with a simple architecture, our model achieves consistently good predictive performance on known and unknown object classes and exhibits a good understanding of the long term dynamics.
KTH Royal Institute of Technology（瑞典皇家理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；Örebro University（厄勒布鲁大学）
Modelling of contact-rich tasks is challenging and cannot be entirely solved using classical control approaches due to the difficulty of constructing an analytic description of the contact dynamics. Additionally, in a manipulation task like food-cutting, purely learning-based methods such as Reinforcement Learning, require either a vast amount of data that is expensive to collect on a real robot, or a highly realistic simulation environment, which is currently not available. This paper presents a data-driven control approach that employs a recurrent neural network to model the dynamics for a M…
To better optimise the global food supply chain, robotic solutions are needed to automate tasks currently completed by humans. Namely, phenotyping, quality analysis and harvesting are all open problems in the field of agricultural robotics. Robotic perception is a key challenge for autonomous solutions to such problems as scene understanding and object detection are vital prerequisites to any grasping tasks that a robot may undertake. This work conducts a brief review of modern robot perception models and discusses their efficacy within the agri-food domain.
The Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Machine Learning Department, Carnegie Mellon University（卡内基梅隆大学机器学习系）
Cutting is a common form of manipulation when working with divisible objects such as food, rope, or clay. Cooking in particular relies heavily on cutting to divide food items into desired shapes. However, cutting food is a challenging task due to the wide range of material properties exhibited by food items. Due to this variability, the same cutting motions cannot be used for all food items. Sensations from contact events, e.g., when placing the knife on the food item, will also vary depending on the material properties, and the robot will need to adapt accordingly. In this paper, we propose u…
A key challenge in robotic food manipulation is modeling the material properties of diverse and deformable food items. We propose using a multimodal sensory approach to interact and play with food that facilitates the ability to distinguish these properties across food items. First, we use a robotic arm and an array of sensors, which are synchronized using ROS, to collect a diverse dataset consisting of 21 unique food items with varying slices and properties. Afterwards, we learn visual embedding networks that utilize a combination of proprioceptive, audio, and visual data to encode similariti…
Agri-Food is the largest manufacturing sector in the UK. It supports a food chain that generates over £108bn p.a., with 3.9m employees in a truly international industry and exports £20bn of UK manufactured goods. However, the global food chain is under pressure from population growth, climate change, political pressures affecting migration, population drift from rural to urban regions and the demographics of an aging global population. These challenges are recognised in the UK Industrial Strategy white paper and backed by significant investment via a Wave 2 Industrial Challenge Fund Investment…
Paul G. Allen School of Computer Science and Engineering, University of Washington（保罗·G·艾伦计算机科学与工程学院，华盛顿大学）
Autonomous feeding is challenging because it requires manipulation of food items with various compliance, sizes, and shapes. To understand how humans manipulate food items during feeding and to explore ways to adapt their strategies to robots, we collected a rich dataset of human trajectories by asking them to pick up food and feed it to a mannequin. From the analysis of the collected haptic and motion signals, we demonstrate that humans adapt their control policies to accommodate to the compliance and shape of the food item being acquired. We propose a taxonomy of manipulation strategies for …
Reshetnev Siberian State University of Science and Technology（列舍特涅夫西伯利亚国立科学技术大学）；Siberian Federal University（西伯利亚联邦大学）
The article considers solving the problem of precision cutting of honeycomb blocks. The urgency of using arbitrary shapes application cutting from honey-comb blocks made of modern composite materials is substantiated. The problem is to obtain a cut of the given shape from honeycomb blocks. The complexity of this problem is in the irregular pattern of honeycomb blocks and the presence of double edges, which forces an operator to scan each block before cutting. It is necessary to take into account such restrictions as the place and angle of the cut and size of the knife, its angle when cutting a…
NVIDIA（英伟达）；University of Southern California（南加州大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Washington（华盛顿大学）；University of Sydney（悉尼大学）
Robotic cutting of soft materials is critical for applications such as food processing, household automation, and surgical manipulation. As in other areas of robotics, simulators can facilitate controller verification, policy learning, and dataset generation. Moreover, differentiable simulators can enable gradient-based optimization, which is invaluable for calibrating simulation parameters and optimizing controllers. In this work, we present DiSECt: the first differentiable simulator for cutting soft materials. The simulator augments the finite element method (FEM) with a continuous contact m…
University of Edinburgh（爱丁堡大学）；Monash University（莫纳什大学）
Dynamic Movement Primitives (DMPs) offer great versatility for encoding, generating and adapting complex end-effector trajectories. DMPs are also very well suited to learning manipulation skills from human demonstration. However, the reactive nature of DMPs restricts their applicability for tool use and object manipulation tasks involving non-holonomic constraints, such as scalpel cutting or catheter steering. In this work, we extend the Cartesian space DMP formulation by adding a coupling term that enforces a pre-defined set of non-holonomic constraints. We obtain the closed-form expression f…
Central South University（中南大学）；Norwegian University of Science & Technology (NTNU)（挪威科技大学）
The use of robotics in food industry is becoming more popular in recent years. The trend seems to continue as long as the robotics technology meets diverse and challenging needs of the food producers. Rapid developments in digital computers and control systems technologies have significant impact in robotics like any other engineering fields. By utilizing new hardware and software tools, design of these complex systems that need strong integration of distinct disciplines is no longer difficult compared to the past. Therefore, the purpose of this paper is to design and implement a micro-control…
Institute for Intelligent Systems Research and Innovation (IISRI)（智能系统研究与创新研究所）；Deakin University（迪肯大学）；Faculty of Health（健康学院）
In robotic surgery, pattern cutting through a deformable material is a challenging research field. The cutting procedure requires a robot to concurrently manipulate a scissor and a gripper to cut through a predefined contour trajectory on the deformable sheet. The gripper ensures the cutting accuracy by nailing a point on the sheet and continuously tensioning the pinch point to different directions while the scissor is in action. The goal is to find a pinch point and a corresponding tensioning policy to minimize damage to the material and increase cutting accuracy measured by the symmetric dif…
DecompGrind: A Decomposition Framework for Robotic Grinding via Cutting-Surface Planning and Contact-Force Adaptation
Nara Institute of Science and Technology (NAIST)（奈良先端科学技术大学院大学）；Keio University（庆应义塾大学）；Kyoto University（京都大学）；Advanced Telecommunications Research Institute International (ATR)（国际电气通信基础技术研究所）
Peduncle Gripping and Cutting Force for Strawberry Harvesting Robotic End-effector Design
Lincoln Institute of Agri-Technology (LIAT)（林肯农业技术研究所）；School of Computer Science（计算机科学学院）；Lincoln Center of Autonomous Systems (LCAS)（林肯自主系统中心）；University of Lincoln（林肯大学）
DexNinja: Learning Robust Dexterous Cutting Policy with a Real-to-Sim-to-Real Data Engine
College of AI, Tsinghua University（清华大学人工智能学院）；INSAIT, Sofia University “St. Kliment Ohridski”（索菲亚大学圣克莱门特奥赫里德斯基分校INSAIT）；Peking University（北京大学）；IIIS, Tsinghua University（清华大学高等研究院）；University of Southern California（南加州大学）；Beijing Institute of Technology（北京理工大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；vivo Mobile Communication Co., Ltd.（维沃移动通信有限公司）
University of Southern California（南加州大学）；NVIDIA（英伟达）；University of Washington（华盛顿大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Sydney（悉尼大学）
Robotic cutting of soft materials is critical for applications such as food processing, household automation, and surgical manipulation. As in other areas of robotics, simulators can facilitate controller verification, policy learning, and dataset generation. Moreover, differentiable simulators can enable gradient-based optimization, which is invaluable for calibrating simulation parameters and optimizing controllers. In this work, we present DiSECt: the first differentiable simulator for cutting soft materials. The simulator augments the finite element method (FEM) with a continuous contact m…
End-to-end example-based sim-to-real RL policy transfer based on neural stylisation with application to robotic cutting
Extreme Robotics Lab, School of Metallurgy and Materials, University of Birmingham（极端机器人实验室、冶金与材料学院、伯明翰大学）；The Faraday Institution, Quad One, Harwell Science and Innovation Campus（法拉第机构、Quad One、哈威尔科学与创新校园）；School of Computer Science and Digital Technologies, Aston University（计算机科学与数字技术学院、阿斯顿大学）
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Most existing robotic harvesters utilize a unimanual approach; a single arm grasps the crop and detaches it, either via a detachment movement, or by cutting its stem with a specially designed gripper/cutter end-effector. However, such unimanual solutions cannot be applied for sensitive crops and cluttered environments like grapes and a vineyard where obstacles may occlude the stem and leave no space for the cutter's placement. In such cases, the solution would require a bimanual robot in order to visually unveil the stem and manipulate the grasped crop to create cutting affordances which is si…
Sashimi-Bot: Autonomous Tri-manual Advanced Manipulation and Cutting of Deformable Objects
Robotics and Intelligent Automation（机器人与智能自动化）；SINTEF Ocean（SINTEF海洋）；Department of Computer Science（计算机科学系）；NTNU - Norwegian University of Science and Technology（挪威科学与技术大学）；QUT Centre for Robotics（昆士兰大学机器人中心）；Queensland University of Technology（昆士兰技术大学）；Inria, University of Rennes, CNRS, IRISA（Inria、里昂大学、法国国家科学研究中心、IRISA）；CSAIL（计算机科学与人工智能实验室）；MIT - Massachusets Institute of Technology（麻省理工学院）；RealTek；Norwegian University of Life Sciences (NMBU)（挪威生命科学大学（NMBU））
Contemporary robots in precision agriculture focus primarily on automated harvesting or remote sensing to monitor crop health. Comparatively less work has been performed with respect to collecting physical leaf samples in the field and retaining them for further analysis. Typically, orchard growers manually collect sample leaves and utilize them for stem water potential measurements to analyze tree health and determine irrigation routines. While this technique benefits orchard management, the process of collecting, assessing, and interpreting measurements requires significant human labor and o…
California Institute of Technology（加州理工学院）；Miso Robotics（味噌机器人公司）
Recent advances allow for the automation of food preparation in high-throughput environments, yet the successful deployment of these robots requires the planning and execution of quick, robust, and ultimately collision-free behaviors. In this work, we showcase a novel framework for modifying previously generated trajectories of robotic manipulators in highly detailed and dynamic collision environments using Control Barrier Functions (CBFs). This method dynamically re-plans previously validated behaviors in the presence of changing environments -- and does so in a computationally efficient mann…
Queensland University of Technology（昆士兰科技大学）；ETH Zurich（苏黎世联邦理工学院）
Using robots to harvest sweet peppers in protected cropping environments has remained unsolved despite considerable effort by the research community over several decades. In this paper, we present the robotic harvester, Harvey, designed for sweet peppers in protected cropping environments that achieved a 76.5% success rate (within a modified scenario) which improves upon our prior work which achieved 58% and related sweet pepper harvesting work which achieved 33\%. This improvement was primarily achieved through the introduction of a novel peduncle segmentation system using an efficient deep c…
UK National Centre for Nuclear Robotics（英国国家核机器人中心）；University of Birmingham（伯明翰大学）；National Nuclear Laboratory（国家核实验室）；University of West England（西英格兰大学）
This paper presents a method for constrained motion planning from vision, which enables a robot to move its end-effector over an observed surface, given start and destination points. The robot has no prior knowledge of the surface shape, but observes it from a noisy point-cloud camera. We consider the multi-objective optimisation problem of finding robot trajectories which maximise the robot's manipulability throughout the motion, while also minimising surface-distance travelled between the two points. This work has application in industrial problems of \textit{rough} robotic cutting, \textit{…
School of Computer Science, McGill University（麦吉尔大学计算机科学学院）；Mila Quebec AI Institute（魁北克人工智能研究所 Mila）；Department of Mechanical Engineering, McGill University（麦吉尔大学机械工程系）
Utilization of latent space to capture a lower-dimensional representation of a complex dynamics model is explored in this work. The targeted application is of a robotic manipulator executing a complex environment interaction task, in particular, cutting a wooden object. We train two flavours of Variational Autoencoders---standard and Vector-Quantised---to learn the latent space which is then used to infer certain properties of the cutting operation, such as whether the robot is cutting or not, as well as, material and geometry of the object being cut. The two VAE models are evaluated with reco…
California Institute of Technology（加州理工学院）；Stanford University（斯坦福大学）
Assistance during eating is essential for those with severe mobility issues or eating risks. However, dependence on traditional human caregivers is linked to malnutrition, weight loss, and low self-esteem. For those who require eating assistance, a semi-autonomous robotic platform can provide independence and a healthier lifestyle. We demonstrate an essential capability of this platform: safe, comfortable, and effective transfer of a bite-sized food item from a utensil directly to the inside of a person's mouth. Our system uses a force-reactive controller to safely accommodate the user's motio…
Safety-Ensured Robotic Control Framework for Cutting Task Automation in Endoscopic Submucosal Dissection
Division of Mechanical System Engineering, Myongji University, Yongin-si, Gyeonggi-do, Republic of Korea（机械系统工程系，明州大学，韩国京畿道 Yongin市）；SDU Robotics, the Maersk Mc-Kinney Moller Institute, University of Southern Denmark, Odense, Denmark（SDU机器人，马士克麦金尼莫勒研究所，丹麦南部大学，Odense）；Department of Mechanical Engineering, Myongji University, Yongin-si, Gyeonggi-do, Republic of Korea（机械工程系，明州大学，韩国京畿道 Yongin市）
Robotic systems in manufacturing applications commonly assume known object geometry and appearance. This simplifies the task for the 3D perception algorithms and allows the manipulation to be more deterministic. However, those approaches are not easily transferable to the agricultural and food domains due to the variability and deformability of natural food. We demonstrate an approach applied to poultry products that allows picking up a whole chicken from an unordered bin using a suction cup gripper, estimating its pose using a Deep Learning approach, and placing it in a canonical orientation …
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins School of Medicine（约翰斯·霍普金斯医学院）
Craniomaxillofacial reconstruction with patient-specific customized craniofacial implants (CCIs) is most commonly performed for large-sized skeletal defects. Because the exact size of skull resection may not be known prior to the surgery, in the single-stage cranioplasty, a large CCI is prefabricated and resized intraoperatively with a manual-cutting process provided by a surgeon. The manual resizing, however, may be inaccurate and significantly add to the operating time. This paper introduces a fast and non-contact approach for intraoperatively determining the exact contour of the skull resec…
Lincoln Institute for Agri-food Technology, University of Lincoln（林肯大学林肯农业食品技术研究所）
Global food security is an issue that is fast becoming a critical matter in the world today. Global warming, climate change and a range of other impacts caused by humans, such as carbon emissions, sociopolitical and economical challenges (e.g. war), traditional workforce/labour decline and population growth are straining global food security. The need for high-speed and reliable wireless communication in agriculture is becoming more of a necessity rather than a technological demonstration or showing superiority in the field. Governments and industries around the world are seeing more urgency i…
Adaptive Depth-Map-Guided Bundle Adjustment for Correspondence-Free Multi-View Point Cloud Registration
Robotics Institute, Faculty of Engineering and Information Technology, University of Technology Sydney（悉尼科技大学工程与信息技术学院机器人研究所）；School of Informatics, The University of Edinburgh（爱丁堡大学信息学院）
Donders Institute for Brain, Cognition, and Behaviour（唐德斯大脑、认知与行为研究所）；Radboud University（拉德堡德大学）；Joint Research Centre, European Commission（欧盟委员会联合研究中心）；Ghent University（根特大学）；imec（微电子研究中心）；Instituto Superio Técnico（高等技术学院）；University of Lisbon（里斯本大学）
Understanding and modelling children's cognitive processes and their behaviour in the context of their interaction with robots and social artificial intelligence systems is a fundamental prerequisite for meaningful and effective robot interventions. However, children's development involve complex faculties such as exploration, creativity and curiosity which are challenging to model. Also, often children express themselves in a playful way which is different from a typical adult behaviour. Different children also have different needs, and it remains a challenge in the current state of the art t…
Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Southern University of Science and Technology（南方科技大学）；South China University of Technology（华南理工大学）；University of Essex（埃塞克斯大学）；Instituto de Telecomunicaões（葡萄牙电信研究所）；University of New South Wales（新南威尔士大学）
Wireless energy transfer (WET) is a ground-breaking technology for cutting the last wire between mobile sensors and power grids in smart cities. Yet, WET only offers effective transmission of energy over a short distance. Robotic WET is an emerging paradigm that mounts the energy transmitter on a mobile robot and navigates the robot through different regions in a large area to charge remote energy harvesters. However, it is challenging to determine the robotic charging strategy in an unknown and dynamic environment due to the uncertainty of obstacles. This paper proposes a hardware-in-the-loop…
