Thin leaves, fine stems, self-occlusion, non-rigid and slowly changing structures make plants difficult for three-dimensional (3D) scanning and reconstruction -- two critical steps in automated visual phenotyping. Many current solutions such as laser scanning, structured light, and multiview stereo can struggle to acquire usable 3D models because of limitations in scanning resolution and calibration accuracy. In response, we have developed a fast, low-cost, 3D scanning platform to image plants on a rotating stage with two tilting DSLR cameras centred on the plant. This uses new methods of camera calibration and background removal to achieve high-accuracy 3D reconstruction. We assessed the system's accuracy using a 3D visual hull reconstruction algorithm applied on 2 plastic models of dicotyledonous plants, 2 sorghum plants and 2 wheat plants across different sets of tilt angles. Scan times ranged from 3 minutes (to capture 72 images using 2 tilt angles), to 30 minutes (to capture 360 images using 10 tilt angles). The leaf lengths, widths, areas and perimeters of the plastic models were measured manually and compared to measurements from the scanning system: results were within 3-4% of each other. The 3D reconstructions obtained with the scanning system show excellent geometric agreement with all six plant specimens, even plants with thin leaves and fine stems.
University of Bonn（波恩大学）；Julius Kühn-Institut, Federal Research Centre for Cultivated Plants（尤利乌斯·库恩研究所（联邦栽培植物研究中心））
Semantic labeling of 3D point clouds is important for the derivation of 3D models from real world scenarios in several economic fields such as building industry, facility management, town planning or heritage conservation. In contrast to these most common applications, we describe in this study the semantic labeling of 3D point clouds derived from plant organs by high-precision scanning. Our approach is optimized for the task of plant phenotyping with its very specific challenges and is employing a deep learning framework. Thereby, we report important experiences concerning detailed parameter …
A Low-Cost Photogrammetry System for 3D Plant Modeling and Phenotyping
Department of Applied Computer Science, University of Winnipeg（应用计算机科学系，温尼伯大学）；Department of Physics, University of Winnipeg（物理系，温尼伯大学）；Department of Computer Science, University of Manitoba（计算机科学系，曼尼托巴大学）；Department of Plant Sciences, University of Saskatchewan（植物科学系，萨斯喀彻温大学）
University of Western Ontario（西安大略大学）；Canadian Surgical Technologies & Advanced Robotics（加拿大外科技术与先进机器人中心）；Stryker Mako Surgical Corporation（史赛克马科外科公司）；Bulgarian Academy of Sciences（保加利亚科学院）；University of Guelph（圭尔夫大学）
Machine vision for plant phenotyping is an emerging research area for producing high throughput in agriculture and crop science applications. Since 2D based approaches have their inherent limitations, 3D plant analysis is becoming state of the art for current phenotyping technologies. We present an automated system for analyzing plant growth in indoor conditions. A gantry robot system is used to perform scanning tasks in an automated manner throughout the lifetime of the plant. A 3D laser scanner mounted as the robot's payload captures the surface point cloud data of the plant from multiple vi…
Exploring Accurate 3D Phenotyping in Greenhouse through Neural Radiance Fields
Guangdong Academic of Agriculture Science（广东省农业科学院）；College of Engineering, South China Agriculture University（华南农业大学工程学院）；Department of Mechanical and Aerospace Engineering, Monash University（莫纳什大学机械与航空航天工程学院）
SPROUT: A Scalable Diffusion Foundation Model for Multi-Crop Plant Phenotyping
Graduate School of Agricultural and Life Sciences, The University of Tokyo（东京大学大学院农学生命科学研究科）；National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学多谱信息智能处理技术全国重点实验室、人工智能与自动化学院）；Institute of Agricultural Machinery, NARO（日本国立研究开发法人农业·食品产业技术综合研究机构农业机械研究所）；Institute of Life and Environmental Sciences, University of Tsukuba（筑波大学生命环境科学研究所）
BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions
Center for Robotics, University of Bonn（柏林洪堡大学机器人中心）；Institute of Sugar Beet Research（甘蔗研究院）；Lamarr Institute for Machine Learning and Artificial Intelligence（Lamarr人工智能与机器学习研究所）
A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond
Department of Electrical and Computer Engineering, Michigan State University（电气与计算机工程系，密歇根州立大学）；Department of Systems and Information Engineering & Link Lab, University of Virginia（系统与信息工程系及Link实验室，弗吉尼亚大学）；Department of Statistics and Probability, Michigan State University（统计与概率系，密歇根州立大学）；School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University（化学、化工与生物技术学院，南洋理工大学）；Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密西西比州立大学）；Department of Mechanical Engineering, Michigan State University（机械工程系，密歇根州立大学）
Object-Centric 3D Gaussian Splatting for Strawberry Plant Reconstruction and Phenotyping
Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI, USA（电气与计算机工程系，密歇根州立大学）；Department of Mechanical Engineering, Michigan State University, East Lansing, MI, USA（机械工程系，密歇根州立大学）；Truck Crops Branch Experiment Station, Mississippi State University, Starkville, MS, USA（卡车作物分支实验站，密苏里州立大学）；Department of Agricultural and Biological Engineering, Mississippi State University, Starkville, MS, USA（农业与生物工程系，密苏里州立大学）；Tandon School of Engineering, New York University, NY, USA（坦顿工程学院，纽约大学）
Plant Phenomics can be used to monitor the health and the growth of plants. Computer vision applications like stereo reconstruction, image retrieval, object tracking, and object recognition play an important role in imaging based plant phenotyping. This paper offers a comparative evaluation of some popular 3D correspondence grouping algorithms, motivated by the important role that they can play in tasks such as model creation, plant recognition and identifying plant parts. Another contribution of this paper is the extension of 2D maximum likelihood matching to 3D Maximum Likelihood Estimation …
In this work, the use of t-SNE is proposed to embed 3D point clouds of plants into 2D space for plant characterization. It is demonstrated that t-SNE operates as a practical tool to flatten and visualize a complete 3D plant model in 2D space. The perplexity parameter of t-SNE allows 2D rendering of plant structures at various organizational levels. Aside from the promise of serving as a visualization tool for plant scientists, t-SNE also provides a gateway for processing 3D point clouds of plants using their embedded counterparts in 2D. In this paper, simple methods were proposed to perform se…
GrowSplat: Constructing Temporal Digital Twins of Plants with Gaussian Splats
The AUTOLab at UC Berkeley（加州大学伯克利分校AUTOLab）；Siemens Research Lab（西门子研究实验室）；Netherlands Plant Eco-phenotyping Centre, Wageningen University and Research（荷兰植物生态表型中心，瓦赫宁根大学与研究中心）
In order to apply the recent successes of machine learning and automated plant phenotyping on a large scale using agricultural robotics, efficient and general algorithms must be designed to intelligently split crop fields into small, yet actionable, portions that can then be processed by more complex algorithms. In this paper, we notice a similarity between the current state-of-the-art for separating corn plants and a commonly used density-based clustering algorithm, Quickshift. Exploiting this similarity we propose a number of novel, application-specific algorithms with the goal of producing …
Indian Institute of Technology（印度理工学院）；The University of Tokyo（东京大学）
In light of growing challenges in agriculture with ever growing food demand across the world, efficient crop management techniques are necessary to increase crop yield. Precision agriculture techniques allow the stakeholders to make effective and customized crop management decisions based on data gathered from monitoring crop environments. Plant phenotyping techniques play a major role in accurate crop monitoring. Advancements in deep learning have made previously difficult phenotyping tasks possible. This survey aims to introduce the reader to the state of the art research in deep plant pheno…
Eskisehir Osmangazi University（埃斯基谢希尔奥斯曼加齐大学）；INRAe（法国国家农业食品与环境研究院）；Institut de Recherche en Horticulture et Semences（园艺与种子研究所）；Université d’Angers（昂热大学）
Segmentation of structural parts of 3D models of plants is an important step for plant phenotyping, especially for monitoring architectural and morphological traits. Current state-of-the art approaches rely on hand-crafted 3D local features for modeling geometric variations in plant structures. While recent advancements in deep learning on point clouds have the potential of extracting relevant local and global characteristics, the scarcity of labeled 3D plant data impedes the exploration of this potential. We adapted six recent point-based deep learning architectures (PointNet, PointNet++, DGC…
Procedural Generation of 3D Maize Plant Architecture from LIDAR Data
Department of Mechanical Engineering, Iowa State University（机械工程系，爱荷华州立大学）；Department of Agronomy, Iowa State University（农业系，爱荷华州立大学）；Translational AI Center, Iowa State University（转化人工智能中心，爱荷华州立大学）；Plant Science Institute, Iowa State University（植物科学研究所，爱荷华州立大学）
Zero-Shot 3D Plant Organ Segmentation with SAM3 and Semantic NeRFs
Fraunhofer Institute for Integrated Circuits (IIS)（弗劳恩霍夫集成电路研究所）；Otto-Friedrich-Universität Bamberg（班贝格奥托·弗里德里希大学）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）
Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council（国家研究委员会先进制造智能工业技术与系统研究所）；University of Salento（萨伦托大学）
Plant phenotyping, that is, the quantitative assessment of plant traits including growth, morphology, physiology, and yield, is a critical aspect towards efficient and effective crop management. Currently, plant phenotyping is a manually intensive and time consuming process, which involves human operators making measurements in the field, based on visual estimates or using hand-held devices. In this work, methods for automated grapevine phenotyping are developed, aiming to canopy volume estimation and bunch detection and counting. It is demonstrated that both measurements can be effectively pe…
Iterative Motion Compensation for Canonical 3D Reconstruction from UAV Plant Images Captured in Windy Conditions
Autonomous Intelligent Systems - Computer Science Institute VI and Center for Robotics, University of Bonn, Germany（自主智能系统 - 计算科学研究所VI和机器人中心，波恩大学，德国）；Lamarr Institute for Machine Learning and Artificial Intelligence, Germany（拉玛尔机器学习与人工智能研究所，德国）
Biomass phenotyping of oilseed rape through UAV multi-view oblique imaging with 3DGS and SAM model
College of Biosystems Engineering and Food Science, Zhejiang University（浙江大学生物系统工程与食品科学学院）；Key Laboratory of Spectroscopy Sensing, Ministry of Agriculture and Rural Affairs（农业农村部光谱传感重点实验室）；College of Agriculture and Biotechnology, Zhejiang University（浙江大学农业与生物技术学院）
Athena Research and Innovation Center（雅典娜研究与创新中心）；Institute of Mediterranean & Forest Ecosystems（地中海与森林生态系统研究所）
Plant phenotyping refers to a quantitative description of the plants properties, however in image-based phenotyping analysis, our focus is primarily on the plants anatomical, ontogenetical and physiological properties.This technique reinforced by the success of Deep Learning in the field of image based analysis is applicable to a wide range of research areas making high-throughput screens of plants possible, reducing the time and effort needed for phenotypic characterization.In this study, we use Deep Learning methods (supervised and unsupervised learning based approaches) to semantically segm…
The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Data61（联邦科学与工业研究组织（CSIRO）Data61）
The traditional paradigm of applying deep learning -- collect, annotate and train on data -- is not applicable to image-based plant phenotyping as almost 400,000 different plant species exists. Data costs include growing physical samples, imaging and labelling them. Model performance is impacted by the species gap between the domain of each plant species, it is not generalisable and may not transfer to unseen plant species. In this paper, we investigate the use of synthetic data for leaf instance segmentation. We study multiple synthetic data training regimes using Mask-RCNN when few or no ann…
College of Information Sciences and Technology, Donghua University（东华大学信息科学与技术学院）；Engineering Research Center of Digitized Textile & Fashion Technology, Ministry of Education, Donghua University（东华大学数字化纺织服装技术教育部工程研究中心）
Automatic leaf segmentation, as well as identification and classification methods that built upon it, are able to provide immediate monitoring for plant growth status to guarantee the output. Although 3D plant point clouds contain abundant phenotypic features, plant leaves are usually distributed in clusters and are sometimes seriously overlapped in the canopy. Therefore, it is still a big challenge to automatically segment each individual leaf from a highly crowded plant canopy in 3D for plant phenotyping purposes. In this work, we propose an overlapping-free individual leaf segmentation meth…
PlantSegNeRF: A few-shot, cross-species method for plant 3D instance point cloud reconstruction via joint-channel NeRF with multi-view image instance matching
Deep learning models have been successfully deployed for a diverse array of image-based plant phenotyping applications including disease detection and classification. However, successful deployment of supervised deep learning models requires large amount of labeled data, which is a significant challenge in plant science (and most biological) domains due to the inherent complexity. Specifically, data annotation is costly, laborious, time consuming and needs domain expertise for phenotyping tasks, especially for diseases. To overcome this challenge, active learning algorithms have been proposed …
PLANesT-3D: A new annotated dataset for segmentation of 3D plant point clouds
Faculty of Agriculture（农业学院）；Usak University（乌萨克大学）；Department of Electrical-Electronics Engineering（电气电子工程系）；Eskisehir Osmangazi University（埃斯基谢普大学）；SADC Department（SADC部门）；Huawei Türkiye R&D Center（华为土耳其研发中心）；Laboratoire Angevin de Recherche en Ingénierie des Systèmes (LARIS)（安热斯工程系统研究实验室）；Université d’Angers（安格斯大学）
Tomato Multi-Angle Multi-Pose Dataset for Fine-Grained Phenotyping
Institute for Breeding Research on Horticultural Crops, Julius Kuehn-Institute（园艺作物育种研究所，朱利叶斯·克恩研究所）；Computer Graphics Group, Center for Sensor Systems (ZESS), University of Siegen（计算机图形组，传感器系统中心（ZESS），希尔德斯海姆大学）
Jeonbuk National University（全北国立大学）；Kyungpook National University（庆北国立大学）
Technology to recognize the type of component represented by a point cloud is required in the reconstruction process of an as-built model of a process plant based on laser scanning. The reconstruction process of a process plant through laser scanning is divided into point cloud registration, point cloud segmentation, and component type recognition and placement. Loss of shape data or imbalance of point cloud density problems generally occur in the point cloud data collected from large-scale facilities. In this study, we experimented with the possibility of applying object recognition technolog…
MaizeField3D: A Curated 3D Point Cloud and Procedural Model Dataset of Field-Grown Maize from a Diversity Panel
Department of Mechanical Engineering, Iowa State University, Ames, USA（机械工程系，爱荷华州立大学）；Translational AI Research and Education Center, Ames, USA（转化人工智能研究与教育中心）；Plant Science Institute, Iowa State University, Ames, USA（植物科学研究所）；Interdepartmental Genetics and Genomics Graduate Program, Iowa State University, Ames, USA（跨部门遗传学和基因组学研究生项目）；Department of Agronomy, Iowa State University, Ames, USA（农业学系）
Unmanned Aerial Vehicles (UAVs) have become popular for use in plant phenotyping of field based crops, such as maize and sorghum, due to their ability to acquire high resolution data over field trials. Field experiments, which may comprise thousands of plants, are planted according to experimental designs to evaluate varieties or management practices. For many types of phenotyping analysis, we examine smaller groups of plants known as "plots." In this paper, we propose a new plot extraction method that will segment a UAV image into plots. We will demonstrate that our method achieves higher plo…
The Alan Turing Institute（艾伦·图灵研究所）；Forschungszentrum Jülich（于利希研究中心）；University of Edinburgh（爱丁堡大学）
In recent years, there has been an increasing interest in image-based plant phenotyping, applying state-of-the-art machine learning approaches to tackle challenging problems, such as leaf segmentation (a multi-instance problem) and counting. Most of these algorithms need labelled data to learn a model for the task at hand. Despite the recent release of a few plant phenotyping datasets, large annotated plant image datasets for the purpose of training deep learning algorithms are lacking. One common approach to alleviate the lack of training data is dataset augmentation. Herein, we propose an al…
Deep learning techniques have been successfully deployed for automating plant stress identification and quantification. In recent years, there is a growing push towards training models that are interpretable -i.e. that justify their classification decisions by visually highlighting image features that were crucial for classification decisions. The expectation is that trained network models utilize image features that mimic visual cues used by plant pathologists. In this work, we compare some of the most popular interpretability methods: Saliency Maps, SmoothGrad, Guided Backpropogation, Deep T…
LeafTrackNet: A Deep Learning Framework for Robust Leaf Tracking in Top-Down Plant Phenotyping
Leibniz Institute for Agricultural Engineering（莱布尼茨农业工程与生物经济研究所）；BASF Belgium Coordination Center CommV（巴斯夫比利时协调中心）；BASF Metabolome Solutions GmbH（巴斯夫代谢组解决方案公司）；Technical University Berlin（柏林技术大学）；University of Potsdam（波茨坦大学）
High-fidelity 3D Reconstruction of Plants using Neural Radiance Field
College of Engineering, South China Agriculture University（华南农业大学工程学院）；Department of Mechanical and Aerospace Engineering, Monash University（莫纳什大学机械与航空航天工程系）
Institute of Imaging & Computer Vision, RWTH Aachen University（亚琛工业大学成像与计算机视觉研究所）；Julius Kühn Institute: Federal Research Centre for Cultivated Plants（尤利乌斯·库恩研究所（联邦作物研究中心））；LemnaTec GmbH（LemnaTec有限公司）
The beet cyst nematode (BCN) Heterodera schachtii is a plant pest responsible for crop loss on a global scale. Here, we introduce a high-throughput system based on computer vision that allows quantifying BCN infestation and characterizing nematode cysts through phenotyping. After recording microscopic images of soil extracts in a standardized setting, an instance segmentation algorithm serves to detect nematode cysts in these samples. Going beyond fast and precise cyst counting, the image-based approach enables quantification of cyst density and phenotyping of morphological features of cysts u…
Iowa State University（爱荷华州立大学）；Plant Science Institute, Iowa State University（爱荷华州立大学植物科学研究所）；Translational AI Research and Education Center, Iowa State University（爱荷华州立大学转化人工智能研究与教育中心）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）
United States Department of Agriculture, Agricultural Research Service（美国农业部农业研究服务局）；Appalachian Fruit Research Station (USDA-ARS-AFRS)（阿巴拉契亚水果研究站（美国农业部农业研究服务局阿巴拉契亚水果研究站））；Universidad Tecnológica de Pereira（佩雷拉理工大学）；San Joaquin Valley Agricultural Sciences Center (USDA-ARS-SJVASC)（圣华金河谷农业科学中心（美国农业部农业研究服务局圣华金河谷农业科学中心））
Images are used frequently in plant phenotyping to capture measurements. This chapter offers a repeatable method for capturing two-dimensional measurements of plant parts in field or laboratory settings using a variety of camera styles (cellular phone, DSLR), with the addition of a printed calibration pattern. The method is based on calibrating the camera using information available from the EXIF tags from the image, as well as visual information from the pattern. Code is provided to implement the method, as well as a dataset for testing. We include steps to verify protocol correctness by imag…
