Wood logs picking is a challenging task to automate. Indeed, logs usually come in cluttered configurations, randomly orientated and overlapping. Recent work on log picking automation usually assume that the logs' pose is known, with little consideration given to the actual perception problem. In this paper, we squarely address the latter, using a data-driven approach. First, we introduce a novel dataset, named TimberSeg 1.0, that is densely annotated, i.e., that includes both bounding boxes and pixel-level mask annotations for logs. This dataset comprises 220 images with 2500 individually segmented logs. Using our dataset, we then compare three neural network architectures on the task of individual logs detection and segmentation; two region-based methods and one attention-based method. Unsurprisingly, our results show that axis-aligned proposals, failing to take into account the directional nature of logs, underperform with 19.03 mAP. A rotation-aware proposal method significantly improve results to 31.83 mAP. More interestingly, a Transformer-based approach, without any inductive bias on rotations, outperformed the two others, achieving a mAP of 57.53 on our dataset. Our use case demonstrates the limitations of region-based approaches for cluttered, elongated objects. It also highlights the potential of attention-based methods on this specific task, as they work directly at the pixel-level. These encouraging results indicate that such a perception system could be used to assist the operators on the short-term, or to fully automate log picking operations in the future.
Towards Autonomous Wood-Log Grasping with a Forestry Crane: Simulator and Benchmarking
Automation & Control Institute (ACIN), TU Wien（自动化与控制研究所（ACIN），维也纳技术大学）；AIT Austrian Institute of Technology GmbH（奥地利技术研究院）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）
In this research, a fully neural network based visual perception framework for autonomous apple harvesting is proposed. The proposed framework includes a multi-function neural network for fruit recognition and a Pointnet grasp estimation to determine the proper grasp pose to guide the robotic execution. Fruit recognition takes raw input of RGB images from the RGB-D camera to perform fruit detection and instance segmentation, and Pointnet grasp estimation take point cloud of each fruit as input and output the prediction of grasp pose for each of fruits. The proposed framework is validated by us…
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Grasping a particular object may require a dedicated grasping movement that may also be specific to the robot end-effector. No generic and autonomous method does exist to generate these movements without making hypotheses on the robot or on the object. Learning methods could help to autonomously discover relevant grasping movements, but they face an important issue: grasping movements are so rare that a learning method based on exploration has little chance to ever observe an interesting movement, thus creating a bootstrap issue. We introduce an approach to generate diverse grasping movements …
University of Waterloo（滑铁卢大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
There has been increasing interest in smart factories powered by robotics systems to tackle repetitive, laborious tasks. One impactful yet challenging task in robotics-powered smart factory applications is robotic grasping: using robotic arms to grasp objects autonomously in different settings. Robotic grasping requires a variety of computer vision tasks such as object detection, segmentation, grasp prediction, pick planning, etc. While significant progress has been made in leveraging of machine learning for robotic grasping, particularly with deep learning, a big challenge remains in the need…
Where to Perch in a Tree: Vision-Guidance for Tree-Grasping Drones
Department of Civil, Aerospace and Design Engineering, University of Bristol（布里斯托大学土木、航空航天与设计工程系）；Laboratory of Sustainability Robotics at Swiss Federal Laboratories for Materials Science and Technology (EMPA)（瑞士材料科学与技术联邦实验室可持续机器人实验室）；Ecole Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
UltraDexGrasp: Learning Universal Dexterous Grasping for Bimanual Robots with Synthetic Data
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；The University of Hong Kong（香港大学）；Peking University（北京大学）
Robotics Lab, Department of Electrical and Computer Engineering, College of Engineering, University of Illinois Chicago（机器人实验室，电气与计算机工程系，工程学院，伊利诺伊大学芝加哥分校）；Surgical Innovation and Training Lab, Department of Surgery, College of Medicine, University of Illinois Chicago（外科创新与培训实验室，外科系，医学院，伊利诺伊大学芝加哥分校）
Florida Institute of Technology（佛罗里达理工学院）；Energy Management Aerospace（能源管理航空航天公司）
Autonomous navigation and path-planning around non-cooperative space objects is an enabling technology for on-orbit servicing and space debris removal systems. The navigation task includes the determination of target object motion, the identification of target object features suitable for grasping, and the identification of collision hazards and other keep-out zones. Given this knowledge, chaser spacecraft can be guided towards capture locations without damaging the target object or without unduly the operations of a servicing target by covering up solar arrays or communication antennas. One w…
Technical University of Munich（慕尼黑工业大学）；Madras Institute of Technology, Anna University（安娜大学马德拉斯理工学院）
A robot operating in unstructured environments must be able to discriminate between different grasping styles depending on the prospective manipulation task. Having a system that allows learning from remote non-expert demonstrations can very feasibly extend the cognitive skills of a robot for task-oriented grasping. We propose a novel two-step framework towards this aim. The first step involves grasp area estimation by segmentation. We receive grasp area demonstrations for a new task via interactive segmentation, and learn from these few demonstrations to estimate the required grasp area on an…
Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly
School of Advanced Manufacturing, Sun Yat-sen University（中山大学先进制造学院）；School of Electronics and Communication Engineering, Sun Yat-sen University（中山大学电子与通信工程学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
Performance Evaluation of Deep Learning for Tree Branch Segmentation in Autonomous Forestry Systems
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（惠灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Canterbury（坎特伯雷大学）
Autonomous Unmanned Aerial Manipulators (UAMs) have shown promising potentials to transform passive sensing missions into active 3-dimension interactive missions, but they still suffer from some difficulties impeding their wide applications, such as target detection and stabilization. This letter presents a vision-based autonomous UAM with a 3DoF robotic arm for rotational grasping, with a compensation on displacement for center of gravity. First, the hardware, software architecture and state estimation methods are detailed. All the mechanical designs are fully provided as open-source hardware…
Vision-based segmentation in forested environments is a key functionality for autonomous forestry operations such as tree felling and forwarding. Deep learning algorithms demonstrate promising results to perform visual tasks such as object detection. However, the supervised learning process of these algorithms requires annotations from a large diversity of images. In this work, we propose to use simulated forest environments to automatically generate 43 k realistic synthetic images with pixel-level annotations, and use it to train deep learning algorithms for tree detection. This allows us to …
This paper presents a novel robot vision architecture for perceiving generic 3D clothes configurations. Our architecture is hierarchically structured, starting from low-level curvatures, across mid-level geometric shapes \& topology descriptions; and finally approaching high-level semantic surface structure descriptions. We demonstrate our robot vision architecture in a customised dual-arm industrial robot with our self-designed, off-the-self stereo vision system, carrying out autonomous grasping and dual-arm flattening. It is worth noting that the proposed dual-arm flattening approach is uniq…
Northeastern University（东北大学）；The Chinese University of Hong Kong（香港中文大学）；Ulster University（阿尔斯特大学）
This paper presents the first active object mapping framework for complex robotic manipulation and autonomous perception tasks. The framework is built on an object SLAM system integrated with a simultaneous multi-object pose estimation process that is optimized for robotic grasping. Aiming to reduce the observation uncertainty on target objects and increase their pose estimation accuracy, we also design an object-driven exploration strategy to guide the object mapping process, enabling autonomous mapping and high-level perception. Combining the mapping module and the exploration strategy, an a…
Progressive Per-Branch Depth Optimization for DEFOM-Stereo and SAM3 Joint Analysis in UAV Forestry Applications
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；Victoria University of Wellington（威灵顿维多利亚大学）；Department of Computer Science and Software Engineering（计算机科学与软件工程系）；University of Canterbury（坎特伯雷大学）
Tree perception is an essential building block toward autonomous forestry operations. Current developments generally consider input data from lidar sensors to solve forest navigation, tree detection and diameter estimation problems. Whereas cameras paired with deep learning algorithms usually address species classification or forest anomaly detection. In either of these cases, data unavailability and forest diversity restrain deep learning developments for autonomous systems. So, we propose two densely annotated image datasets - 43k synthetic, 100 real - for bounding box, segmentation mask and…
NVIDIA（英伟达）；German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）；University of Washington（华盛顿大学）
Grasping unseen objects in unconstrained, cluttered environments is an essential skill for autonomous robotic manipulation. Despite recent progress in full 6-DoF grasp learning, existing approaches often consist of complex sequential pipelines that possess several potential failure points and run-times unsuitable for closed-loop grasping. Therefore, we propose an end-to-end network that efficiently generates a distribution of 6-DoF parallel-jaw grasps directly from a depth recording of a scene. Our novel grasp representation treats 3D points of the recorded point cloud as potential grasp conta…
The ability to grasp ordinary and potentially never-seen objects is an important feature in both domestic and industrial robotics. For a system to accomplish this, it must autonomously identify grasping locations by using information from various sensors, such as Microsoft Kinect 3D camera. Despite numerous progress, significant work still remains to be done in this field. To this effect, we propose a dictionary learning and sparse representation (DLSR) framework for representing RGBD images from 3D sensors in the context of determining such good grasping locations. In contrast to previously p…
Optimizing Grasping in Legged Robots: A Deep Learning Approach to Loco-Manipulation
College of Mechanical Engineering, Federal University of Uberlândia（联邦大学伯南迪亚学院机械工程学院）；Department of Mechanical Engineering, University of São Paulo（圣保罗大学机械工程系）
