Selectively picking a target fruit surrounded by obstacles is one of the major challenges for fruit harvesting robots. Different from traditional obstacle avoidance methods, this paper presents an active obstacle separation strategy that combines push and drag motions. The separation motion and trajectory are generated based on the 3D visual perception of the obstacle information around the target. A linear push is used to clear the obstacles from the area below the target, while a zig-zag push that contains several linear motions is proposed to push aside more dense obstacles. The zig-zag push can generate multi-directional pushes and the side-to-side motion can break the static contact force between the target and obstacles, thus helping the gripper to receive a target in more complex situations. Moreover, we propose a novel drag operation to address the issue of mis-capturing obstacles located above the target, in which the gripper drags the target to a place with fewer obstacles and then pushes back to move the obstacles aside for further detachment. Furthermore, an image processing pipeline consisting of color thresholding, object detection using deep learning and point cloud operation, is developed to implement the proposed method on a harvesting robot. Field tests show that the proposed method can improve the picking performance substantially. This method helps to enable complex clusters of fruits to be harvested with a higher success rate than conventional methods.
In the robotic crop harvesting environment, foreign objects intrusion in the gripper workspace is frequently occurring and unignorable, however, rarely addressed. This paper presents a novel intelligent robotic grasping method capable of handling obstacle interference, which is the first of its kind in the literature. The proposed method combines deep learning algorithms with low-cost tactile sensing hardware on a multi-DoF soft robotic gripper. Through experimental validations, the proposed method demonstrated promising performance in distinguishing various grasping scenarios. The 4-finger in…
Hierarchical Tri-manual Planning for Vision-assisted Fruit Harvesting with Quadrupedal Robots
Institute for Integrative & Innovative Research (I 3 R), University of Arkansas, USA（整合与创新研究所（I 3 R）、亚拉巴马大学）；Department of Electrical and Computer Engineering, University of California, Riverside, USA（电气与计算机工程系、加州大学河滨分校）
Accurate depth-sensing plays a crucial role in securing a high success rate of robotic harvesting in natural orchard environments. Solid-state LiDAR (SSL), a recently introduced LiDAR technique, can perceive high-resolution geometric information of the scenes, which can be potential utilised to receive accurate depth information. Meanwhile, the fusion of the sensory information from LiDAR and camera can significantly enhance the sensing ability of the harvesting robots. This work introduces a LiDAR-camera fusion-based visual sensing and perception strategy to perform accurate fruit localisatio…
Field robotic harvesting is a promising technique in recent development of agricultural industry. It is vital for robots to recognise and localise fruits before the harvesting in natural orchards. However, the workspace of harvesting robots in orchards is complex: many fruits are occluded by branches and leaves. It is important to estimate a proper grasping pose for each fruit before performing the manipulation. In this study, a geometry-aware network, A3N, is proposed to perform end-to-end instance segmentation and grasping estimation using both color and geometry sensory data from a RGB-D ca…
HarvestPoint-ACT: Explicit Target Selection and Harvest-Point Conditioning for Robotic Fruit Harvesting under Occlusion
College of Computer and Information Sciences, Fujian Agriculture and Forestry University（福建农林大学计算机与信息学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Graduate School of Engineering Science, The University of Osaka（大阪大学工程科学研究生院）
Beijing Academy of Agriculture and Forestry Sciences（北京市农林科学院）；Jiangsu University（江苏大学）
The emergence of harvesting robotics offers a promising solution to the issue of limited agricultural labor resources and the increasing demand for fruits. Despite notable advancements in the field of harvesting robotics, the utilization of such technology in orchards is still limited. The key challenge is to improve operational efficiency. Taking into account inner-arm conflicts, couplings of DoFs, and dynamic tasks, we propose a task planning strategy for a harvesting robot with four arms in this paper. The proposed method employs a Markov game framework to formulate the four-arm robotic har…
University of Lincoln（林肯大学）；Centre for Autonomous Systems (L-CAS)（自主系统中心（L-CAS））
Robotic technology is increasingly considered the major mean for fruit picking. However, picking fruits in a dense cluster imposes a challenging research question in terms of motion/path planning as conventional planning approaches may not find collision-free movements for the robot to reach-and-pick a ripe fruit within a dense cluster. In such cases, the robot needs to safely push unripe fruits to reach a ripe one. Nonetheless, existing approaches to planning pushing movements in cluttered environments either are computationally expensive or only deal with 2-D cases and are not suitable for f…
Georgia Institute of Technology（佐治亚理工学院）；Georgia Institute of Technology/Emory（佐治亚理工学院/埃默里大学）
Growing global demand for food, coupled with continuing labor shortages, motivates the need for automated agricultural harvesting. While some specialty crops (e.g., apples, peaches, blueberries) can be harvested via existing harvesting modalities, fruits such as blackberries and raspberries require delicate handling to mitigate fruit damage that could significantly impact marketability. This motivates the development of soft robotic solutions that enable efficient, delicate harvesting. This paper presents the design, fabrication, and feasibility testing of a tendon-driven soft gripping system …
Optimal Scheduling of a Dual-Arm Robot for Efficient Strawberry Harvesting in Plant Factories
ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou, China（浙大杭州国际科技成果转化中心，浙江大学，杭州，中国）；Department of Mechanical and Aerospace Engineering, University of California, Davis（加州大学戴维斯分校机械与航空航天工程系）；Department of Biological and Agricultural Engineering, University of California, Davis（加州大学戴维斯分校生物与农业工程系）；College Of Biosystems Engineering And Food Science, Zhejiang University, Hangzhou, China（浙江大学生物系统工程与食品科学学院）
The Australian Center for Field Robotics（澳大利亚现场机器人中心）；The University of Sydney（悉尼大学）
The human labour required for tree crop harvesting is a major cost component in fruit production and is increasing. To address this, many existing research works have sought to demonstrate commercially viable robotic harvesting for tree crops, though successful commercial products resulting from these have been few and far between. Systems developed for specific crops such as sweet peppers or apples have shown promise, but the vast majority of cultivar types remain unaddressed and developing a specific system for each one is inefficient. In this work a flexible and modular development platform…
Vision-Based Obstacle Separation for Strawberry Harvesting in Clusters Using Hierarchical Reinforcement Learning
The Intelligent Equipment Research Center, Beijing Academy of Agriculture and Forestry Sciences（北京市农林科学院智能装备研究中心）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Happy Elements Ltd.（乐元素科技有限公司）
GS-NBV: a Geometry-based, Semantics-aware Viewpoint Planning Algorithm for Avocado Harvesting under Occlusions
Autonomous Robots and Control Systems Lab, University of California, Riverside（加州大学河滨分校自主机器人与控制系统实验室）；Perception, Robotics, AI, Sensing Lab, University of Colorado, Boulder（科罗拉多大学博尔德分校感知、机器人、人工智能与传感实验室）
The Australian Center for Field Robotics（澳大利亚野外机器人中心）；The University of Sydney（悉尼大学）
Many automated operations in agriculture, such as weeding and plant counting, require robust and accurate object detectors. Robotic fruit harvesting is one of these, and is an important technology to address the increasing labour shortages and uncertainty suffered by tree crop growers. An eye-in-hand sensing setup is commonly used in harvesting systems and provides benefits to sensing accuracy and flexibility. However, as the hand and camera move from viewing the entire trellis to picking a specific fruit, large changes in lighting, colour, obscuration and exposure occur. Object detection algo…
Fast Heuristic Scheduling and Trajectory Planning for Robotic Fruit Harvesters with Multiple Cartesian Arms
Department of Mechanical & Aerospace Engineering University of California, Davis（机械与航空航天工程系加州大学戴维斯分校）；Department of Biological and Agricultural Engineering University of California, Davis（生物与农业工程系加州大学戴维斯分校）
Mechanizing the manual harvesting of fresh market fruits constitutes one of the biggest challenges to the sustainability of the fruit industry. During manual harvesting of some fresh-market crops like strawberries and table grapes, pickers spend significant amounts of time walking to carry full trays to a collection station at the edge of the field. A step toward increasing harvest automation for such crops is to deploy harvest-aid collaborative robots (co-bots) that transport the empty and full trays, thus increasing harvest efficiency by reducing pickers' non-productive walking times. This w…
Mechanizing the manual harvesting of fresh market fruits constitutes one of the biggest challenges to the sustainability of the fruit industry. During manual harvesting of some fresh-market crops like strawberries and table grapes, pickers spend significant amounts of time walking to carry full trays to a collection station at the edge of the field. A step toward increasing harvest automation for such crops is to deploy harvest-aid collaborative robots (co-bots) that transport the empty and full trays, thus increasing harvest efficiency by reducing pickers' non-productive walking times. This w…
Michigan State University（密歇根州立大学）；United States Department of Agriculture Agricultural Research Service（美国农业部农业研究服务局）
There is a growing need for robotic apple harvesting due to decreasing availability and rising cost in labor. Towards the goal of developing a viable robotic system for apple harvesting, this paper presents synergistic mechatronic design and motion control of a robotic apple harvesting prototype, which lays a critical foundation for future advancements. Specifically, we develop a deep learning-based fruit detection and localization system using an RGB-D camera. A three degree-of-freedom manipulator is then designed with a hybrid pneumatic/motor actuation mechanism to achieve fast and dexterous…
Optimising robotic operation speed with edge computing over 5G networks: Insights from selective harvesting robots
Lincoln Institute of Agri-food Technology, University of Lincoln（林肯大学林肯农食品技术学院）；School of Computer Science, University of Lincoln（林肯大学计算机科学学院）；School of Computer Science and Electronic Engineering, University of Surrey（萨里大学计算机科学与电子工程学院）
Lincoln Institute for Agri-food Technology（林肯农业食品技术研究院）；University of Lincoln（林肯大学）；Kings College London（伦敦国王学院）
Challenges in strawberry picking made selective harvesting robotic technology demanding. However, selective harvesting of strawberries is complicated forming a few scientific research questions. Most available solutions only deal with a specific picking scenario, e.g., picking only a single variety of fruit in isolation. Nonetheless, most economically viable (e.g. high-yielding and/or disease-resistant) varieties of strawberry are grown in dense clusters. The current perception technology in such use cases is inefficient. In this work, we developed a novel system capable of harvesting strawber…
Advancement and Field Evaluation of a Dual-arm Apple Harvesting Robot
Department of Mechanical Engineering（机械工程系）；Michigan State University（密歇根州立大学）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Comp. Sci. and Engineering（计算机科学与工程系）；United States Department of Agriculture Agricultural Research Service（美国农业部农业研究服务）
On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions
Dept. of Electrical and Computer Engineering, Hellenic Mediterranean University（电气与计算机工程系，希腊地中海大学）；Dept. of Mechanical Engineering, Hellenic Mediterranean University（机械工程系，希腊地中海大学）；Institute of Computer Science, Foundation for Research and Technology - Hellas（计算机科学研究所，希腊基础研究与技术基金会）
Investigating Sensors and Methods in Grasp State Classification in Agricultural Manipulation
Department of Mechanical Engineering, University of Illinois at Urbana-Champaign（机械工程系，伊利诺伊大学厄巴纳-香槟分校）；Department of Industrial and Enterprise Systems Engineering, University of Illinois at Urbana-Champaign（工业与企业系统工程系，伊利诺伊大学厄巴纳-香槟分校）
Michigan State University（密歇根州立大学）；United States Department of Agriculture Agricultural Research Service（美国农业部农业研究局）
Due to labor shortage and rising labor cost for the apple industry, there is an urgent need for the development of robotic systems to efficiently and autonomously harvest apples. In this paper, we present a system overview and algorithm design of our recently developed robotic apple harvester prototype. Our robotic system is enabled by the close integration of several core modules, including visual perception, planning, and control. This paper covers the main methods and advancements in deep learning-based multi-view fruit detection and localization, unified picking and dropping planning, and …
The problem of robotic lime picking is challenging; lime plants have dense foliage which makes it difficult for a robotic arm to grasp a lime without coming in contact with leaves. Existing approaches either do not consider leaves, or treat them as obstacles and completely avoid them, often resulting in undesirable or infeasible plans. We focus on reaching a lime in the presence of dense foliage by considering the leaves of a plant as 'permeable obstacles' with a collision cost. We then adapt the rapidly exploring random tree star (RRT*) algorithm for the problem of fruit harvesting by incorpo…
