Deep learning techniques have been successfully deployed for automating plant stress identification and quantification. In recent years, there is a growing push towards training models that are interpretable -i.e. that justify their classification decisions by visually highlighting image features that were crucial for classification decisions. The expectation is that trained network models utilize image features that mimic visual cues used by plant pathologists. In this work, we compare some of the most popular interpretability methods: Saliency Maps, SmoothGrad, Guided Backpropogation, Deep Taylor Decomposition, Integrated Gradients, Layer-wise Relevance Propagation and Gradient times Input, for interpreting the deep learning model. We train a DenseNet-121 network for the classification of eight different soybean stresses (biotic and abiotic). Using a dataset consisting of 16,573 RGB images of healthy and stressed soybean leaflets captured under controlled conditions, we obtained an overall classification accuracy of 95.05 \%. For a diverse subset of the test data, we compared the important features with those identified by a human expert. We observed that most interpretability methods identify the infected regions of the leaf as important features for some -- but not all -- of the correctly classified images. For some images, the output of the interpretability methods indicated that spurious feature correlations may have been used to correctly classify them. Although the output explanation maps of these interpretability methods may be different from each other for a given image, we advocate the use of these interpretability methods as `hypothesis generation' mechanisms that can drive scientific insight.
Indian Institute of Technology（印度理工学院）；The University of Tokyo（东京大学）
In light of growing challenges in agriculture with ever growing food demand across the world, efficient crop management techniques are necessary to increase crop yield. Precision agriculture techniques allow the stakeholders to make effective and customized crop management decisions based on data gathered from monitoring crop environments. Plant phenotyping techniques play a major role in accurate crop monitoring. Advancements in deep learning have made previously difficult phenotyping tasks possible. This survey aims to introduce the reader to the state of the art research in deep plant pheno…
University of Bonn（波恩大学）；Julius Kühn-Institut, Federal Research Centre for Cultivated Plants（尤利乌斯·库恩研究所（联邦栽培植物研究中心））
Semantic labeling of 3D point clouds is important for the derivation of 3D models from real world scenarios in several economic fields such as building industry, facility management, town planning or heritage conservation. In contrast to these most common applications, we describe in this study the semantic labeling of 3D point clouds derived from plant organs by high-precision scanning. Our approach is optimized for the task of plant phenotyping with its very specific challenges and is employing a deep learning framework. Thereby, we report important experiences concerning detailed parameter …
A Low-Cost Photogrammetry System for 3D Plant Modeling and Phenotyping
Department of Applied Computer Science, University of Winnipeg（应用计算机科学系，温尼伯大学）；Department of Physics, University of Winnipeg（物理系，温尼伯大学）；Department of Computer Science, University of Manitoba（计算机科学系，曼尼托巴大学）；Department of Plant Sciences, University of Saskatchewan（植物科学系，萨斯喀彻温大学）
University of Western Ontario（西安大略大学）；Canadian Surgical Technologies & Advanced Robotics（加拿大外科技术与先进机器人中心）；Stryker Mako Surgical Corporation（史赛克马科外科公司）；Bulgarian Academy of Sciences（保加利亚科学院）；University of Guelph（圭尔夫大学）
Machine vision for plant phenotyping is an emerging research area for producing high throughput in agriculture and crop science applications. Since 2D based approaches have their inherent limitations, 3D plant analysis is becoming state of the art for current phenotyping technologies. We present an automated system for analyzing plant growth in indoor conditions. A gantry robot system is used to perform scanning tasks in an automated manner throughout the lifetime of the plant. A 3D laser scanner mounted as the robot's payload captures the surface point cloud data of the plant from multiple vi…
Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council（国家研究委员会先进制造智能工业技术与系统研究所）；University of Salento（萨伦托大学）
Plant phenotyping, that is, the quantitative assessment of plant traits including growth, morphology, physiology, and yield, is a critical aspect towards efficient and effective crop management. Currently, plant phenotyping is a manually intensive and time consuming process, which involves human operators making measurements in the field, based on visual estimates or using hand-held devices. In this work, methods for automated grapevine phenotyping are developed, aiming to canopy volume estimation and bunch detection and counting. It is demonstrated that both measurements can be effectively pe…
Exploring Accurate 3D Phenotyping in Greenhouse through Neural Radiance Fields
Guangdong Academic of Agriculture Science（广东省农业科学院）；College of Engineering, South China Agriculture University（华南农业大学工程学院）；Department of Mechanical and Aerospace Engineering, Monash University（莫纳什大学机械与航空航天工程学院）
Athena Research and Innovation Center（雅典娜研究与创新中心）；Institute of Mediterranean & Forest Ecosystems（地中海与森林生态系统研究所）
Plant phenotyping refers to a quantitative description of the plants properties, however in image-based phenotyping analysis, our focus is primarily on the plants anatomical, ontogenetical and physiological properties.This technique reinforced by the success of Deep Learning in the field of image based analysis is applicable to a wide range of research areas making high-throughput screens of plants possible, reducing the time and effort needed for phenotypic characterization.In this study, we use Deep Learning methods (supervised and unsupervised learning based approaches) to semantically segm…
The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Data61（联邦科学与工业研究组织（CSIRO）Data61）
The traditional paradigm of applying deep learning -- collect, annotate and train on data -- is not applicable to image-based plant phenotyping as almost 400,000 different plant species exists. Data costs include growing physical samples, imaging and labelling them. Model performance is impacted by the species gap between the domain of each plant species, it is not generalisable and may not transfer to unseen plant species. In this paper, we investigate the use of synthetic data for leaf instance segmentation. We study multiple synthetic data training regimes using Mask-RCNN when few or no ann…
Australian National University（澳大利亚国立大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；Queensland University of Technology（昆士兰科技大学）
Thin leaves, fine stems, self-occlusion, non-rigid and slowly changing structures make plants difficult for three-dimensional (3D) scanning and reconstruction -- two critical steps in automated visual phenotyping. Many current solutions such as laser scanning, structured light, and multiview stereo can struggle to acquire usable 3D models because of limitations in scanning resolution and calibration accuracy. In response, we have developed a fast, low-cost, 3D scanning platform to image plants on a rotating stage with two tilting DSLR cameras centred on the plant. This uses new methods of came…
SPROUT: A Scalable Diffusion Foundation Model for Multi-Crop Plant Phenotyping
Graduate School of Agricultural and Life Sciences, The University of Tokyo（东京大学大学院农学生命科学研究科）；National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学多谱信息智能处理技术全国重点实验室、人工智能与自动化学院）；Institute of Agricultural Machinery, NARO（日本国立研究开发法人农业·食品产业技术综合研究机构农业机械研究所）；Institute of Life and Environmental Sciences, University of Tsukuba（筑波大学生命环境科学研究所）
Deep learning models have been successfully deployed for a diverse array of image-based plant phenotyping applications including disease detection and classification. However, successful deployment of supervised deep learning models requires large amount of labeled data, which is a significant challenge in plant science (and most biological) domains due to the inherent complexity. Specifically, data annotation is costly, laborious, time consuming and needs domain expertise for phenotyping tasks, especially for diseases. To overcome this challenge, active learning algorithms have been proposed …
Tomato Multi-Angle Multi-Pose Dataset for Fine-Grained Phenotyping
Institute for Breeding Research on Horticultural Crops, Julius Kuehn-Institute（园艺作物育种研究所，朱利叶斯·克恩研究所）；Computer Graphics Group, Center for Sensor Systems (ZESS), University of Siegen（计算机图形组，传感器系统中心（ZESS），希尔德斯海姆大学）
BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions
Center for Robotics, University of Bonn（柏林洪堡大学机器人中心）；Institute of Sugar Beet Research（甘蔗研究院）；Lamarr Institute for Machine Learning and Artificial Intelligence（Lamarr人工智能与机器学习研究所）
Unmanned Aerial Vehicles (UAVs) have become popular for use in plant phenotyping of field based crops, such as maize and sorghum, due to their ability to acquire high resolution data over field trials. Field experiments, which may comprise thousands of plants, are planted according to experimental designs to evaluate varieties or management practices. For many types of phenotyping analysis, we examine smaller groups of plants known as "plots." In this paper, we propose a new plot extraction method that will segment a UAV image into plots. We will demonstrate that our method achieves higher plo…
The Alan Turing Institute（艾伦·图灵研究所）；Forschungszentrum Jülich（于利希研究中心）；University of Edinburgh（爱丁堡大学）
In recent years, there has been an increasing interest in image-based plant phenotyping, applying state-of-the-art machine learning approaches to tackle challenging problems, such as leaf segmentation (a multi-instance problem) and counting. Most of these algorithms need labelled data to learn a model for the task at hand. Despite the recent release of a few plant phenotyping datasets, large annotated plant image datasets for the purpose of training deep learning algorithms are lacking. One common approach to alleviate the lack of training data is dataset augmentation. Herein, we propose an al…
A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond
Department of Electrical and Computer Engineering, Michigan State University（电气与计算机工程系，密歇根州立大学）；Department of Systems and Information Engineering & Link Lab, University of Virginia（系统与信息工程系及Link实验室，弗吉尼亚大学）；Department of Statistics and Probability, Michigan State University（统计与概率系，密歇根州立大学）；School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University（化学、化工与生物技术学院，南洋理工大学）；Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密西西比州立大学）；Department of Mechanical Engineering, Michigan State University（机械工程系，密歇根州立大学）
LeafTrackNet: A Deep Learning Framework for Robust Leaf Tracking in Top-Down Plant Phenotyping
Leibniz Institute for Agricultural Engineering（莱布尼茨农业工程与生物经济研究所）；BASF Belgium Coordination Center CommV（巴斯夫比利时协调中心）；BASF Metabolome Solutions GmbH（巴斯夫代谢组解决方案公司）；Technical University Berlin（柏林技术大学）；University of Potsdam（波茨坦大学）
Object-Centric 3D Gaussian Splatting for Strawberry Plant Reconstruction and Phenotyping
Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI, USA（电气与计算机工程系，密歇根州立大学）；Department of Mechanical Engineering, Michigan State University, East Lansing, MI, USA（机械工程系，密歇根州立大学）；Truck Crops Branch Experiment Station, Mississippi State University, Starkville, MS, USA（卡车作物分支实验站，密苏里州立大学）；Department of Agricultural and Biological Engineering, Mississippi State University, Starkville, MS, USA（农业与生物工程系，密苏里州立大学）；Tandon School of Engineering, New York University, NY, USA（坦顿工程学院，纽约大学）
Institute of Imaging & Computer Vision, RWTH Aachen University（亚琛工业大学成像与计算机视觉研究所）；Julius Kühn Institute: Federal Research Centre for Cultivated Plants（尤利乌斯·库恩研究所（联邦作物研究中心））；LemnaTec GmbH（LemnaTec有限公司）
The beet cyst nematode (BCN) Heterodera schachtii is a plant pest responsible for crop loss on a global scale. Here, we introduce a high-throughput system based on computer vision that allows quantifying BCN infestation and characterizing nematode cysts through phenotyping. After recording microscopic images of soil extracts in a standardized setting, an instance segmentation algorithm serves to detect nematode cysts in these samples. Going beyond fast and precise cyst counting, the image-based approach enables quantification of cyst density and phenotyping of morphological features of cysts u…
Iowa State University（爱荷华州立大学）；Plant Science Institute, Iowa State University（爱荷华州立大学植物科学研究所）；Translational AI Research and Education Center, Iowa State University（爱荷华州立大学转化人工智能研究与教育中心）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）
United States Department of Agriculture, Agricultural Research Service（美国农业部农业研究服务局）；Appalachian Fruit Research Station (USDA-ARS-AFRS)（阿巴拉契亚水果研究站（美国农业部农业研究服务局阿巴拉契亚水果研究站））；Universidad Tecnológica de Pereira（佩雷拉理工大学）；San Joaquin Valley Agricultural Sciences Center (USDA-ARS-SJVASC)（圣华金河谷农业科学中心（美国农业部农业研究服务局圣华金河谷农业科学中心））
Images are used frequently in plant phenotyping to capture measurements. This chapter offers a repeatable method for capturing two-dimensional measurements of plant parts in field or laboratory settings using a variety of camera styles (cellular phone, DSLR), with the addition of a printed calibration pattern. The method is based on calibrating the camera using information available from the EXIF tags from the image, as well as visual information from the pattern. Code is provided to implement the method, as well as a dataset for testing. We include steps to verify protocol correctness by imag…
From UAV Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping
University of Memphis（孟菲斯大学）；University of Missouri（密苏里大学）；University of Arkansas Division of Agriculture（阿肯色大学农业分部）；Mississippi State University（密西西比州立大学）
Department of Computer Science IV, University of Bonn（波恩大学计算机科学四系）；Institute for Grapevine Breeding Geilweilerhof, Julius Kühn-Institut, Federal Research Centre for Cultivated Plants（联邦栽培植物研究中心尤利乌斯·库恩研究所葡萄育种研究所）
In (grapevine) breeding programs and research, periodic phenotyping and multi-year monitoring of different grapevine traits, like growth or yield, is needed especially in the field. This demand imply objective, precise and automated methods using sensors and adaptive software. This work presents a proof-of-concept analyzing RGB images of different growth stages of grapevines with the aim to detect and quantify promising plant organs which are related to yield. The input images are segmented by a Fully Convolutional Neural Network (FCN) into object and background pixels. The objects are plant o…
Generating high-quality 3DMPCs by adaptive data acquisition and NeREF-based radiometric calibration with UGV plant phenotyping system
College of Biosystems Engineering and Food Science, Zhejiang University（浙江大学生物系统工程与食品科学学院）；Key Laboratory of Spectroscopy Sensing, Ministry of Agriculture and Rural Affairs（农业农村部光谱传感重点实验室）
Indian Institute of Technology Bombay（孟买印度理工学院）；ICAR-Indian Agricultural Research Institute（ICAR-印度农业研究所）
Development of either drought-resistant or drought-tolerant varieties in rice (Oryza sativa L.), especially for high yield in the context of climate change, is a crucial task across the world. The need for high yielding rice varieties is a prime concern for developing nations like India, China, and other Asian-African countries where rice is a primary staple food. The present investigation is carried out for discriminating drought tolerant, and susceptible genotypes. A total of 150 genotypes were grown under controlled conditions to evaluate at High Throughput Plant Phenomics facility, Nanaji …
ChronoRoot 2.0: An Open AI-Powered Platform for 2D Temporal Plant Phenotyping
Instituto de Fisiología, Biología Molecular y Neurociencias (IFIBYNE), CONICET-Universidad de Buenos Aires（生理学、生物分子与神经科学研究所（IFIBYNE），CONICET-布宜诺斯艾利斯大学）；APOLO Biotech（APOLO生物科技）；Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional sinc(i)（信号、系统与计算智能研究所）；CONICET-Universidad Nacional del Litoral（CONICET-沿海大学）；Université Paris-Saclay, CNRS, INRAE, Université Evry, Institute of Plant Sciences Paris-Saclay (IPS2)（巴黎萨克雷大学，CNRS，INRAE，埃夫里大学，巴黎萨克雷植物科学研究所）；Université Paris Cité, CNRS, INRAE, Institute of Plant Sciences Paris-Saclay (IPS2)（巴黎城市大学，CNRS，INRAE，巴黎萨克雷植物科学研究所）
In many agricultural applications one wants to characterize physical properties of plants and use the measurements to predict, for example biomass and environmental influence. This process is known as phenotyping. Traditional collection of phenotypic information is labor-intensive and time-consuming. Use of imagery is becoming popular for phenotyping. In this paper, we present methods to estimate traits of sorghum plants from RBG cameras on board of an unmanned aerial vehicle (UAV). The position and orientation of the imagery together with the coordinates of sparse points along the area of int…
Accurately phenotyping plant wilting is important for understanding responses to environmental stress. Analysis of the shape of plants can potentially be used to accurately quantify the degree of wilting. Plant shape analysis can be enhanced by locating the stem, which serves as a consistent reference point during wilting. In this paper, we show that deep learning methods can accurately segment tomato plant stems. We also propose a control-point-based ground truth method that drastically reduces the resources needed to create a training dataset for a deep learning approach. Experimental result…
GrowSplat: Constructing Temporal Digital Twins of Plants with Gaussian Splats
The AUTOLab at UC Berkeley（加州大学伯克利分校AUTOLab）；Siemens Research Lab（西门子研究实验室）；Netherlands Plant Eco-phenotyping Centre, Wageningen University and Research（荷兰植物生态表型中心，瓦赫宁根大学与研究中心）
Supervised learning is often used to count objects in images, but for counting small, densely located objects, the required image annotations are burdensome to collect. Counting plant organs for image-based plant phenotyping falls within this category. Object counting in plant images is further challenged by having plant image datasets with significant domain shift due to different experimental conditions, e.g. applying an annotated dataset of indoor plant images for use on outdoor images, or on a different plant species. In this paper, we propose a domain-adversarial learning approach for dom…
In order to apply the recent successes of machine learning and automated plant phenotyping on a large scale using agricultural robotics, efficient and general algorithms must be designed to intelligently split crop fields into small, yet actionable, portions that can then be processed by more complex algorithms. In this paper, we notice a similarity between the current state-of-the-art for separating corn plants and a commonly used density-based clustering algorithm, Quickshift. Exploiting this similarity we propose a number of novel, application-specific algorithms with the goal of producing …
Plant phenotyping focuses on the measurement of plant characteristics throughout the growing season, typically with the goal of evaluating genotypes for plant breeding. Estimating plant location is important for identifying genotypes which have low emergence, which is also related to the environment and management practices such as fertilizer applications. The goal of this paper is to investigate methods that estimate plant locations for a field-based crop using RGB aerial images captured using Unmanned Aerial Vehicles (UAVs). Deep learning approaches provide promising capability for locating …
Huazhong University of Science and Technology（华中科技大学）；The University of Adelaide（阿德莱德大学）
Accurately counting maize tassels is important for monitoring the growth status of maize plants. This tedious task, however, is still mainly done by manual efforts. In the context of modern plant phenotyping, automating this task is required to meet the need of large-scale analysis of genotype and phenotype. In recent years, computer vision technologies have experienced a significant breakthrough due to the emergence of large-scale datasets and increased computational resources. Naturally image-based approaches have also received much attention in plant-related studies. Yet a fact is that most…
We present an approach to leaf level segmentation of images of Arabidopsis thaliana plants based upon detected edges. We introduce a novel approach to edge classification, which forms an important part of a method to both count the leaves and establish the leaf area of a growing plant from images obtained in a high-throughput phenotyping system. Our technique uses a relatively shallow convolutional neural network to classify image edges as background, plant edge, leaf-on-leaf edge or internal leaf noise. The edges themselves were found using the Canny edge detector and the classified edges can…
Saint Louis University（圣路易斯大学）；University of Arizona（亚利桑那大学）；Donald Danforth Plant Science Center（唐纳德·丹福斯植物科学中心）
Automated high throughput plant phenotyping involves leveraging sensors, such as RGB, thermal and hyperspectral cameras (among others), to make large scale and rapid measurements of the physical properties of plants for the purpose of better understanding the difference between crops and facilitating rapid plant breeding programs. One of the most basic phenotyping tasks is to determine the cultivar, or species, in a particular sensor product. This simple phenotype can be used to detect errors in planting and to learn the most differentiating features between cultivars. It is also a challenging…
Towards scalable organ level 3D plant segmentation: Bridging the data algorithm computing gap
Department of Biological and Environmental Engineering（生物与环境工程系）；Horticulture Section, School of Integrative Plant Science（植物科学整合学院园艺系）；School of Electrical and Computer Engineering（电气与计算机工程学院）
Counting plant organs such as heads or tassels from outdoor imagery is a popular benchmark computer vision task in plant phenotyping, which has been previously investigated in the literature using state-of-the-art supervised deep learning techniques. However, the annotation of organs in field images is time-consuming and prone to errors. In this paper, we propose a fully unsupervised technique for counting dense objects such as plant organs. We use a convolutional network-based unsupervised segmentation method followed by two post-hoc optimization steps. The proposed technique is shown to prov…
Eskisehir Osmangazi University（埃斯基谢希尔奥斯曼加齐大学）；INRAe（法国国家农业食品与环境研究院）；Institut de Recherche en Horticulture et Semences（园艺与种子研究所）；Université d’Angers（昂热大学）
Segmentation of structural parts of 3D models of plants is an important step for plant phenotyping, especially for monitoring architectural and morphological traits. Current state-of-the art approaches rely on hand-crafted 3D local features for modeling geometric variations in plant structures. While recent advancements in deep learning on point clouds have the potential of extracting relevant local and global characteristics, the scarcity of labeled 3D plant data impedes the exploration of this potential. We adapted six recent point-based deep learning architectures (PointNet, PointNet++, DGC…
Deakin University（迪肯大学）；Lancaster University（兰卡斯特大学）；The University of Warwick（华威大学）；South China University of Technology（华南理工大学）；Medigrowth Australia Pty Ltd（澳大利亚Medigrowth私人有限公司）
As an essential prerequisite task in image-based plant phenotyping, leaf segmentation has garnered increasing attention in recent years. While self-supervised learning is emerging as an effective alternative to various computer vision tasks, its adaptation for image-based plant phenotyping remains rather unexplored. In this work, we present a self-supervised leaf segmentation framework consisting of a self-supervised semantic segmentation model, a color-based leaf segmentation algorithm, and a self-supervised color correction model. The self-supervised semantic segmentation model groups the se…
Iowa State University（爱荷华州立大学）；Syngenta Seeds（先正达种子公司）
The future landscape of modern farming and plant breeding is rapidly changing due to the complex needs of our society. The explosion of collectable data has started a revolution in agriculture to the point where innovation must occur. To a commercial organization, the accurate and efficient collection of information is necessary to ensure that optimal decisions are made at key points of the breeding cycle. However, due to the shear size of a breeding program and current resource limitations, the ability to collect precise data on individual plants is not possible. In particular, efficient phen…
