Segmentation of structural parts of rosebush plants with 3D point-based deep learning methods
Eskisehir Osmangazi University（埃斯基谢希尔奥斯曼加齐大学）；INRAe（法国国家农业食品与环境研究院）；Institut de Recherche en Horticulture et Semences（园艺与种子研究所）；Université d’Angers（昂热大学）
Kaya Turgut, Helin Dutagaci, Gilles Galopin, David Rousseau
Segmentation of structural parts of 3D models of plants is an important step for plant phenotyping, especially for monitoring architectural and morphological traits. Current state-of-the art approaches rely on hand-crafted 3D local features for modeling geometric variations in plant structures. While recent advancements in deep learning on point clouds have the potential of extracting relevant local and global characteristics, the scarcity of labeled 3D plant data impedes the exploration of this potential. We adapted six recent point-based deep learning architectures (PointNet, PointNet++, DGCNN, PointCNN, ShellNet, RIConv) for segmentation of structural parts of rosebush models. We generated 3D synthetic rosebush models to provide adequate amount of labeled data for modification and pre-training of these architectures. To evaluate their performance on real rosebush plants, we used the ROSE-X data set of fully annotated point cloud models. We provided experiments with and without the incorporation of synthetic data to demonstrate the potential of point-based deep learning techniques even with limited labeled data of real plants. The experimental results show that PointNet++ produces the highest segmentation accuracy among the six point-based deep learning methods. The advantage of PointNet++ is that it provides a flexibility in the scales of the hierarchical organization of the point cloud data. Pre-training with synthetic 3D models boosted the performance of all architectures, except for PointNet.
In this work, the use of t-SNE is proposed to embed 3D point clouds of plants into 2D space for plant characterization. It is demonstrated that t-SNE operates as a practical tool to flatten and visualize a complete 3D plant model in 2D space. The perplexity parameter of t-SNE allows 2D rendering of plant structures at various organizational levels. Aside from the promise of serving as a visualization tool for plant scientists, t-SNE also provides a gateway for processing 3D point clouds of plants using their embedded counterparts in 2D. In this paper, simple methods were proposed to perform se…
PlantSegNeRF: A few-shot, cross-species method for plant 3D instance point cloud reconstruction via joint-channel NeRF with multi-view image instance matching
College of Information Sciences and Technology, Donghua University（东华大学信息科学与技术学院）；Engineering Research Center of Digitized Textile & Fashion Technology, Ministry of Education, Donghua University（东华大学数字化纺织服装技术教育部工程研究中心）
Automatic leaf segmentation, as well as identification and classification methods that built upon it, are able to provide immediate monitoring for plant growth status to guarantee the output. Although 3D plant point clouds contain abundant phenotypic features, plant leaves are usually distributed in clusters and are sometimes seriously overlapped in the canopy. Therefore, it is still a big challenge to automatically segment each individual leaf from a highly crowded plant canopy in 3D for plant phenotyping purposes. In this work, we propose an overlapping-free individual leaf segmentation meth…
Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs
Fraunhofer Institute for Integrated Circuits (IIS)（弗劳恩霍夫集成电路研究所）；Otto-Friedrich-Universität Bamberg（班贝格奥托·弗里德里希大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Reliable and automated 3D plant shoot segmentation is a core prerequisite for the extraction of plant phenotypic traits at the organ level. Combining deep learning and point clouds can provide effective ways to address the challenge. However, fully supervised deep learning methods require datasets to be point-wise annotated, which is extremely expensive and time-consuming. In our work, we proposed a novel weakly supervised framework, Eff-3DPSeg, for 3D plant shoot segmentation. First, high-resolution point clouds of soybean were reconstructed using a low-cost photogrammetry system, and the Mes…
PLANesT-3D: A new annotated dataset for segmentation of 3D plant point clouds
Faculty of Agriculture（农业学院）；Usak University（乌萨克大学）；Department of Electrical-Electronics Engineering（电气电子工程系）；Eskisehir Osmangazi University（埃斯基谢普大学）；SADC Department（SADC部门）；Huawei Türkiye R&D Center（华为土耳其研发中心）；Laboratoire Angevin de Recherche en Ingénierie des Systèmes (LARIS)（安热斯工程系统研究实验室）；Université d’Angers（安格斯大学）
Department of Electrical Engineering, Indian Institute of Technology, Delhi（印度理工学院德里分校电气工程系）；Central Electronics Engineering Research Institute, Pilani（中央电子工程研究所，皮拉尼）
Quantification of physiological changes in plants can capture different drought mechanisms and assist in selection of tolerant varieties in a high throughput manner. In this context, an accurate 3D model of plant canopy provides a reliable representation for drought stress characterization in contrast to using 2D images. In this paper, we propose a novel end-to-end pipeline including 3D reconstruction, segmentation and feature extraction, leveraging deep neural networks at various stages, for drought stress study. To overcome the high degree of self-similarities and self-occlusions in plant ca…
College of Information and Electrical Engineering, Shenyang Agricultural University（沈阳农业大学信息与电气工程学院）；School of Information and Intelligence Engineering, University of Sanya（三亚学院信息与智能工程学院）
Nowadays, there are many approaches to acquire three-dimensional (3D) point clouds of maize plants. However, automatic stem-leaf segmentation of maize shoots from three-dimensional (3D) point clouds remains challenging, especially for new emerging leaves that are very close and wrapped together during the seedling stage. To address this issue, we propose an automatic segmentation method consisting of three main steps: skeleton extraction, coarse segmentation based on the skeleton, fine segmentation based on stem-leaf classification. The segmentation method was tested on 30 maize seedlings and …
PlantSeg: A Large-Scale In-the-wild Dataset for Plant Disease Segmentation
The University of Queensland（昆士兰大学）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学电气工程与计算机科学学院）；School of Agriculture and Food Sustainability, The University of Queensland（昆士兰大学农业与食品可持续发展学院）；CSIRO（联邦科学与工业研究组织）
We describe a new measure for the evaluation of region level segmentation of objects, as applied to evaluating the accuracy of leaf-level segmentation of plant images. The proposed approach enforces the rule that a region (e.g. a leaf) in either the image being evaluated or the ground truth image evaluated against can be mapped to no more than one region in the other image. We call this measure the subset-matched Jaccard index.
Osaka University（大阪大学）；Japan Science and Technology Agency（日本科学技术振兴机构）
This paper proposes a multi-view extension of instance segmentation without relying on texture or shape descriptor matching. Multi-view instance segmentation becomes challenging for scenes with repetitive textures and shapes, e.g., plant leaves, due to the difficulty of multi-view matching using texture or shape descriptors. To this end, we propose a multi-view region matching method based on epipolar geometry, which does not rely on any feature descriptors. We further show that the epipolar region matching can be easily integrated into instance segmentation and effective for instance-wise 3D …
Accurately phenotyping plant wilting is important for understanding responses to environmental stress. Analysis of the shape of plants can potentially be used to accurately quantify the degree of wilting. Plant shape analysis can be enhanced by locating the stem, which serves as a consistent reference point during wilting. In this paper, we show that deep learning methods can accurately segment tomato plant stems. We also propose a control-point-based ground truth method that drastically reduces the resources needed to create a training dataset for a deep learning approach. Experimental result…
This paper proposes a novel framework for fluorescence plant video processing. The plant research community is interested in the leaf-level photosynthetic analysis within a plant. A prerequisite for such analysis is to segment all leaves, estimate their structures, and track them over time. We identify this as a joint multi-leaf segmentation, alignment, and tracking problem. First, leaf segmentation and alignment are applied on the last frame of a plant video to find a number of well-aligned leaf candidates. Second, leaf tracking is applied on the remaining frames with leaf candidate transform…
SPROUT: A Scalable Diffusion Foundation Model for Multi-Crop Plant Phenotyping
Graduate School of Agricultural and Life Sciences, The University of Tokyo（东京大学大学院农学生命科学研究科）；National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学多谱信息智能处理技术全国重点实验室、人工智能与自动化学院）；Institute of Agricultural Machinery, NARO（日本国立研究开发法人农业·食品产业技术综合研究机构农业机械研究所）；Institute of Life and Environmental Sciences, University of Tsukuba（筑波大学生命环境科学研究所）
Hierarchical Approach for Joint Semantic, Plant Instance, and Leaf Instance Segmentation in the Agricultural Domain
University of Bonn（波恩大学）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；La Sapienza University of Rome（罗马第一大学）
Jeonbuk National University（全北国立大学）；Kyungpook National University（庆北国立大学）
Technology to recognize the type of component represented by a point cloud is required in the reconstruction process of an as-built model of a process plant based on laser scanning. The reconstruction process of a process plant through laser scanning is divided into point cloud registration, point cloud segmentation, and component type recognition and placement. Loss of shape data or imbalance of point cloud density problems generally occur in the point cloud data collected from large-scale facilities. In this study, we experimented with the possibility of applying object recognition technolog…
University of Bonn（波恩大学）；FZ Jülich GmbH（于利希研究中心有限公司）
Magnetic resonance imaging (MRI) enables plant scientists to non-invasively study root system development and root-soil interaction. Challenging recording conditions, such as low resolution and a high level of noise hamper the performance of traditional root extraction algorithms, though. We propose to increase signal-to-noise ratio and resolution by segmenting the scanned volumes into root and soil in super-resolution using a 3D U-Net. Tests on real data show that the trained network is capable to detect most roots successfully and even finds roots that were missed by human annotators. Our ex…
We present an approach to leaf level segmentation of images of Arabidopsis thaliana plants based upon detected edges. We introduce a novel approach to edge classification, which forms an important part of a method to both count the leaves and establish the leaf area of a growing plant from images obtained in a high-throughput phenotyping system. Our technique uses a relatively shallow convolutional neural network to classify image edges as background, plant edge, leaf-on-leaf edge or internal leaf noise. The edges themselves were found using the Canny edge detector and the classified edges can…
BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions
Center for Robotics, University of Bonn（柏林洪堡大学机器人中心）；Institute of Sugar Beet Research（甘蔗研究院）；Lamarr Institute for Machine Learning and Artificial Intelligence（Lamarr人工智能与机器学习研究所）
In order to apply the recent successes of machine learning and automated plant phenotyping on a large scale using agricultural robotics, efficient and general algorithms must be designed to intelligently split crop fields into small, yet actionable, portions that can then be processed by more complex algorithms. In this paper, we notice a similarity between the current state-of-the-art for separating corn plants and a commonly used density-based clustering algorithm, Quickshift. Exploiting this similarity we propose a number of novel, application-specific algorithms with the goal of producing …
Analyzing plant roots is crucial to understand plant performance in different soil environments. While magnetic resonance imaging (MRI) can be used to obtain 3D images of plant roots, extracting the root structural model is challenging due to highly noisy soil environments and low-resolution of MRI images. To improve both contrast and resolution, we adapt the state-of-the-art method RefineNet for 3D segmentation of the plant root MRI images in super-resolution. The networks are trained from few manual segmentations that are augmented by geometric transformations, realistic noise, and other var…
Robotics and Autonomous Systems The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Data61 Brisbane, Australia（澳大利亚联邦科学与工业研究组织（CSIRO）数据61部布里斯班机器人与自主系统部（Robotics and Autonomous Systems））
Automated segmentation of individual leaves of a plant in an image is a prerequisite to measure more complex phenotypic traits in high-throughput phenotyping. Applying state-of-the-art machine learning approaches to tackle leaf instance segmentation requires a large amount of manually annotated training data. Currently, the benchmark datasets for leaf segmentation contain only a few hundred labeled training images. In this paper, we propose a framework for leaf instance segmentation by augmenting real plant datasets with generated synthetic images of plants inspired by domain randomisation. We…
MaizeField3D: A Curated 3D Point Cloud and Procedural Model Dataset of Field-Grown Maize from a Diversity Panel
Department of Mechanical Engineering, Iowa State University, Ames, USA（机械工程系，爱荷华州立大学）；Translational AI Research and Education Center, Ames, USA（转化人工智能研究与教育中心）；Plant Science Institute, Iowa State University, Ames, USA（植物科学研究所）；Interdepartmental Genetics and Genomics Graduate Program, Iowa State University, Ames, USA（跨部门遗传学和基因组学研究生项目）；Department of Agronomy, Iowa State University, Ames, USA（农业学系）
Plant Doctor: A hybrid machine learning and image segmentation software to quantify plant damage in video footage
Waseda University（早稻田大学）；Kyoto Prefecture Momoyama High School（京都府立桃山高等学校）；Kyushu University（九州大学）；The University of Tokyo Hospital（东京大学医学部附属医院）；Ryukoku University（龙谷大学）
Towards scalable organ level 3D plant segmentation: Bridging the data algorithm computing gap
Department of Biological and Environmental Engineering（生物与环境工程系）；Horticulture Section, School of Integrative Plant Science（植物科学整合学院园艺系）；School of Electrical and Computer Engineering（电气与计算机工程学院）
Analysis of Plant Nutrient Deficiencies Using Multi-Spectral Imaging and Optimized Segmentation Model
organization= Temasek Life Sciences Laboratory , country= Singapore；Disruptive \& Sustainable Technologies for Agricultural Precision, Singapore-MIT Alliance for Research；organization= Department of Biological Sciences, National University of Singapore , country= Singapore；organization= Department of Physics, University of Gothenburg , country= Sweden；organization= Science for Life Laboratory, Department of Physics, University of Gothenburg , country= Sweden
Data-driven sensor interpretation in autonomous driving can lead to highly implausible predictions as can most of the time be verified with common-sense knowledge. However, learning common knowledge only from data is hard and approaches for knowledge integration are an active research area. We propose to use a partly human-designed, partly learned set of rules to describe relations between objects of a traffic scene on a high level of abstraction. In doing so, we improve and robustify existing deep neural networks consuming low-level sensor information. We present an initial study adapting the…
Athena Research and Innovation Center（雅典娜研究与创新中心）；Institute of Mediterranean & Forest Ecosystems（地中海与森林生态系统研究所）
Plant phenotyping refers to a quantitative description of the plants properties, however in image-based phenotyping analysis, our focus is primarily on the plants anatomical, ontogenetical and physiological properties.This technique reinforced by the success of Deep Learning in the field of image based analysis is applicable to a wide range of research areas making high-throughput screens of plants possible, reducing the time and effort needed for phenotypic characterization.In this study, we use Deep Learning methods (supervised and unsupervised learning based approaches) to semantically segm…
BMW Group（宝马集团）；Alpen-Adria Universität（阿尔卑斯-亚德里亚大学）
The digital factory provides undoubtedly a great potential for future production systems in terms of efficiency and effectivity. A key aspect on the way to realize the digital copy of a real factory is the understanding of complex indoor environments on the basis of 3D data. In order to generate an accurate factory model including the major components, i.e. building parts, product assets and process details, the 3D data collected during digitalization can be processed with advanced methods of deep learning. In this work, we propose a fully Bayesian and an approximate Bayesian neural network fo…
Counting plant organs such as heads or tassels from outdoor imagery is a popular benchmark computer vision task in plant phenotyping, which has been previously investigated in the literature using state-of-the-art supervised deep learning techniques. However, the annotation of organs in field images is time-consuming and prone to errors. In this paper, we propose a fully unsupervised technique for counting dense objects such as plant organs. We use a convolutional network-based unsupervised segmentation method followed by two post-hoc optimization steps. The proposed technique is shown to prov…
Deep learning techniques involving image processing and data analysis are constantly evolving. Many domains adapt these techniques for object segmentation, instantiation and classification. Recently, agricultural industries adopted those techniques in order to bring automation to farmers around the globe. One analysis procedure required for automatic visual inspection in this domain is leaf count and segmentation. Collecting labeled data from field crops and greenhouses is a complicated task due to the large variety of crops, growth seasons, climate changes, phenotype diversity, and more, espe…
Fruit size and leaf area are important indicators for plant health and are of interest for plant nutrient management, plant protection and harvest. In this research, an image-based method for measuring the fruit volume as well as the leaf area for cabbage is presented. For this purpose, a mask region-based convolutional neural network (Mask R-CNN) was trained to segment the cabbage fruit from the leaves and assign it to the corresponding plant. The results indicated that even with a single camera, the developed method can provide a calculation accuracy of fruit size of 92.6% and an accuracy of…
Istituto Italiano di Tecnologia（意大利理工学院）；Università di Genoa（热那亚大学）；NVIDIA AI Technology Center（英伟达人工智能技术中心）；Università Cattolica del Sacro Cuore（圣心天主教大学）
Grapevine winter pruning is a complex task, that requires skilled workers to execute it correctly. The complexity of this task is also the reason why it is time consuming. Considering that this operation takes about 80-120 hours/ha to be completed, and therefore is even more crucial in large-size vineyards, an automated system can help to speed up the process. To this end, this paper presents a novel multidisciplinary approach that tackles this challenging task by performing object segmentation on grapevine images, used to create a representative model of the grapevine plants. Second, a set of…
The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Data61（联邦科学与工业研究组织（CSIRO）Data61）
The traditional paradigm of applying deep learning -- collect, annotate and train on data -- is not applicable to image-based plant phenotyping as almost 400,000 different plant species exists. Data costs include growing physical samples, imaging and labelling them. Model performance is impacted by the species gap between the domain of each plant species, it is not generalisable and may not transfer to unseen plant species. In this paper, we investigate the use of synthetic data for leaf instance segmentation. We study multiple synthetic data training regimes using Mask-RCNN when few or no ann…
Brazilian Synchrotron Light Laboratory, Brazilian Center for Research in Energy and Materials（巴西能源与材料研究中心巴西同步辐射光实验室）；California Institute of Technology（加州理工学院）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Campinas（坎皮纳斯大学）
The quantitative analysis of 3D confocal microscopy images of the shoot apical meristem helps understanding the growth process of some plants. Cell segmentation in these images is crucial for computational plant analysis and many automated methods have been proposed. However, variations in signal intensity across the image mitigate the effectiveness of those approaches with no easy way for user correction. We propose a web-based collaborative 3D image segmentation application, SEGMENT3D, to leverage automatic segmentation results. The image is divided into 3D tiles that can be either segmented…
