The traditional paradigm of applying deep learning -- collect, annotate and train on data -- is not applicable to image-based plant phenotyping as almost 400,000 different plant species exists. Data costs include growing physical samples, imaging and labelling them. Model performance is impacted by the species gap between the domain of each plant species, it is not generalisable and may not transfer to unseen plant species. In this paper, we investigate the use of synthetic data for leaf instance segmentation. We study multiple synthetic data training regimes using Mask-RCNN when few or no annotated real data is available. We also present UPGen: a Universal Plant Generator for bridging the species gap. UPGen leverages domain randomisation to produce widely distributed data samples and models stochastic biological variation. Our methods outperform standard practices, such as transfer learning from publicly available plant data, by 26.6% and 51.46% on two unseen plant species respectively. We benchmark UPGen by competing in the CVPPP Leaf Segmentation Challenge and set a new state-of-the-art, a mean of 88% across A1-4 test datasets. This study is applicable to use of synthetic data for automating the measurement of phenotypic traits. Our synthetic dataset and pretrained model are available at https://csiro-robotics.github.io/UPGen_Webpage/.
Athena Research and Innovation Center（雅典娜研究与创新中心）；Institute of Mediterranean & Forest Ecosystems（地中海与森林生态系统研究所）
Plant phenotyping refers to a quantitative description of the plants properties, however in image-based phenotyping analysis, our focus is primarily on the plants anatomical, ontogenetical and physiological properties.This technique reinforced by the success of Deep Learning in the field of image based analysis is applicable to a wide range of research areas making high-throughput screens of plants possible, reducing the time and effort needed for phenotypic characterization.In this study, we use Deep Learning methods (supervised and unsupervised learning based approaches) to semantically segm…
Tomato Multi-Angle Multi-Pose Dataset for Fine-Grained Phenotyping
Institute for Breeding Research on Horticultural Crops, Julius Kuehn-Institute（园艺作物育种研究所，朱利叶斯·克恩研究所）；Computer Graphics Group, Center for Sensor Systems (ZESS), University of Siegen（计算机图形组，传感器系统中心（ZESS），希尔德斯海姆大学）
Institute of Imaging & Computer Vision, RWTH Aachen University（亚琛工业大学成像与计算机视觉研究所）；Julius Kühn Institute: Federal Research Centre for Cultivated Plants（尤利乌斯·库恩研究所（联邦作物研究中心））；LemnaTec GmbH（LemnaTec有限公司）
The beet cyst nematode (BCN) Heterodera schachtii is a plant pest responsible for crop loss on a global scale. Here, we introduce a high-throughput system based on computer vision that allows quantifying BCN infestation and characterizing nematode cysts through phenotyping. After recording microscopic images of soil extracts in a standardized setting, an instance segmentation algorithm serves to detect nematode cysts in these samples. Going beyond fast and precise cyst counting, the image-based approach enables quantification of cyst density and phenotyping of morphological features of cysts u…
Department of Computer Science IV, University of Bonn（波恩大学计算机科学四系）；Institute for Grapevine Breeding Geilweilerhof, Julius Kühn-Institut, Federal Research Centre for Cultivated Plants（联邦栽培植物研究中心尤利乌斯·库恩研究所葡萄育种研究所）
In (grapevine) breeding programs and research, periodic phenotyping and multi-year monitoring of different grapevine traits, like growth or yield, is needed especially in the field. This demand imply objective, precise and automated methods using sensors and adaptive software. This work presents a proof-of-concept analyzing RGB images of different growth stages of grapevines with the aim to detect and quantify promising plant organs which are related to yield. The input images are segmented by a Fully Convolutional Neural Network (FCN) into object and background pixels. The objects are plant o…
Indian Institute of Technology（印度理工学院）；The University of Tokyo（东京大学）
In light of growing challenges in agriculture with ever growing food demand across the world, efficient crop management techniques are necessary to increase crop yield. Precision agriculture techniques allow the stakeholders to make effective and customized crop management decisions based on data gathered from monitoring crop environments. Plant phenotyping techniques play a major role in accurate crop monitoring. Advancements in deep learning have made previously difficult phenotyping tasks possible. This survey aims to introduce the reader to the state of the art research in deep plant pheno…
Towards scalable organ level 3D plant segmentation: Bridging the data algorithm computing gap
Department of Biological and Environmental Engineering（生物与环境工程系）；Horticulture Section, School of Integrative Plant Science（植物科学整合学院园艺系）；School of Electrical and Computer Engineering（电气与计算机工程学院）
Multi-growth stage plant recognition: a case study of Palmer amaranth (Amaranthus palmeri) in cotton (Gossypium hirsutum)
School of Life and Environmental Sciences, The University of Sydney（悉尼大学生命与环境科学学院）；Department of Soil and Crop Sciences, Texas A&M University（德克萨斯农工大学土壤与作物科学学院）
University of Bonn（波恩大学）；Julius Kühn-Institut, Federal Research Centre for Cultivated Plants（尤利乌斯·库恩研究所（联邦栽培植物研究中心））
Semantic labeling of 3D point clouds is important for the derivation of 3D models from real world scenarios in several economic fields such as building industry, facility management, town planning or heritage conservation. In contrast to these most common applications, we describe in this study the semantic labeling of 3D point clouds derived from plant organs by high-precision scanning. Our approach is optimized for the task of plant phenotyping with its very specific challenges and is employing a deep learning framework. Thereby, we report important experiences concerning detailed parameter …
Hierarchical Approach for Joint Semantic, Plant Instance, and Leaf Instance Segmentation in the Agricultural Domain
University of Bonn（波恩大学）；University of Oxford（牛津大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；La Sapienza University of Rome（罗马第一大学）
A Low-Cost Photogrammetry System for 3D Plant Modeling and Phenotyping
Department of Applied Computer Science, University of Winnipeg（应用计算机科学系，温尼伯大学）；Department of Physics, University of Winnipeg（物理系，温尼伯大学）；Department of Computer Science, University of Manitoba（计算机科学系，曼尼托巴大学）；Department of Plant Sciences, University of Saskatchewan（植物科学系，萨斯喀彻温大学）
University of Western Ontario（西安大略大学）；Canadian Surgical Technologies & Advanced Robotics（加拿大外科技术与先进机器人中心）；Stryker Mako Surgical Corporation（史赛克马科外科公司）；Bulgarian Academy of Sciences（保加利亚科学院）；University of Guelph（圭尔夫大学）
Machine vision for plant phenotyping is an emerging research area for producing high throughput in agriculture and crop science applications. Since 2D based approaches have their inherent limitations, 3D plant analysis is becoming state of the art for current phenotyping technologies. We present an automated system for analyzing plant growth in indoor conditions. A gantry robot system is used to perform scanning tasks in an automated manner throughout the lifetime of the plant. A 3D laser scanner mounted as the robot's payload captures the surface point cloud data of the plant from multiple vi…
Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council（国家研究委员会先进制造智能工业技术与系统研究所）；University of Salento（萨伦托大学）
Plant phenotyping, that is, the quantitative assessment of plant traits including growth, morphology, physiology, and yield, is a critical aspect towards efficient and effective crop management. Currently, plant phenotyping is a manually intensive and time consuming process, which involves human operators making measurements in the field, based on visual estimates or using hand-held devices. In this work, methods for automated grapevine phenotyping are developed, aiming to canopy volume estimation and bunch detection and counting. It is demonstrated that both measurements can be effectively pe…
Robotics and Autonomous Systems The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Data61 Brisbane, Australia（澳大利亚联邦科学与工业研究组织（CSIRO）数据61部布里斯班机器人与自主系统部（Robotics and Autonomous Systems））
Automated segmentation of individual leaves of a plant in an image is a prerequisite to measure more complex phenotypic traits in high-throughput phenotyping. Applying state-of-the-art machine learning approaches to tackle leaf instance segmentation requires a large amount of manually annotated training data. Currently, the benchmark datasets for leaf segmentation contain only a few hundred labeled training images. In this paper, we propose a framework for leaf instance segmentation by augmenting real plant datasets with generated synthetic images of plants inspired by domain randomisation. We…
Exploring Accurate 3D Phenotyping in Greenhouse through Neural Radiance Fields
Guangdong Academic of Agriculture Science（广东省农业科学院）；College of Engineering, South China Agriculture University（华南农业大学工程学院）；Department of Mechanical and Aerospace Engineering, Monash University（莫纳什大学机械与航空航天工程学院）
Australian National University（澳大利亚国立大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；Queensland University of Technology（昆士兰科技大学）
Thin leaves, fine stems, self-occlusion, non-rigid and slowly changing structures make plants difficult for three-dimensional (3D) scanning and reconstruction -- two critical steps in automated visual phenotyping. Many current solutions such as laser scanning, structured light, and multiview stereo can struggle to acquire usable 3D models because of limitations in scanning resolution and calibration accuracy. In response, we have developed a fast, low-cost, 3D scanning platform to image plants on a rotating stage with two tilting DSLR cameras centred on the plant. This uses new methods of came…
University of Florida（佛罗里达大学）；UniLaSalle Bauvais（UniLaSalle 博韦）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Missouri（密苏里大学）；University of Maine（缅因大学）；Argonne National Laboratory（阿贡国家实验室）
Understanding a plant's root system architecture (RSA) is crucial for a variety of plant science problem domains including sustainability and climate adaptation. Minirhizotron (MR) technology is a widely-used approach for phenotyping RSA non-destructively by capturing root imagery over time. Precisely segmenting roots from the soil in MR imagery is a critical step in studying RSA features. In this paper, we introduce a large-scale dataset of plant root images captured by MR technology. In total, there are over 72K RGB root images across six different species including cotton, papaya, peanut, s…
StomaD2: An All-in-One System for Intelligent Stomatal Phenotype Analysis via Diffusion-Based Restoration Detection Network
Henan Engineering Research Center for Artificial Intelligence Theory and Algorithms（河南人工智能理论与算法工程研究中心）；School of Mathematics and Statistics（数学与统计学学院）；International Joint Research Laboratory for Global Change Ecology（全球变化生态学联合研究实验室）；School of Life Sciences（生命科学学院）；State Key Laboratory of Cotton Biology（棉花生物学国家重点实验室）
SPROUT: A Scalable Diffusion Foundation Model for Multi-Crop Plant Phenotyping
Graduate School of Agricultural and Life Sciences, The University of Tokyo（东京大学大学院农学生命科学研究科）；National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学多谱信息智能处理技术全国重点实验室、人工智能与自动化学院）；Institute of Agricultural Machinery, NARO（日本国立研究开发法人农业·食品产业技术综合研究机构农业机械研究所）；Institute of Life and Environmental Sciences, University of Tsukuba（筑波大学生命环境科学研究所）
Deep learning models have been successfully deployed for a diverse array of image-based plant phenotyping applications including disease detection and classification. However, successful deployment of supervised deep learning models requires large amount of labeled data, which is a significant challenge in plant science (and most biological) domains due to the inherent complexity. Specifically, data annotation is costly, laborious, time consuming and needs domain expertise for phenotyping tasks, especially for diseases. To overcome this challenge, active learning algorithms have been proposed …
BonnBeetClouds3D: A Dataset Towards Point Cloud-based Organ-level Phenotyping of Sugar Beet Plants under Field Conditions
Center for Robotics, University of Bonn（柏林洪堡大学机器人中心）；Institute of Sugar Beet Research（甘蔗研究院）；Lamarr Institute for Machine Learning and Artificial Intelligence（Lamarr人工智能与机器学习研究所）
Unmanned Aerial Vehicles (UAVs) have become popular for use in plant phenotyping of field based crops, such as maize and sorghum, due to their ability to acquire high resolution data over field trials. Field experiments, which may comprise thousands of plants, are planted according to experimental designs to evaluate varieties or management practices. For many types of phenotyping analysis, we examine smaller groups of plants known as "plots." In this paper, we propose a new plot extraction method that will segment a UAV image into plots. We will demonstrate that our method achieves higher plo…
The Alan Turing Institute（艾伦·图灵研究所）；Forschungszentrum Jülich（于利希研究中心）；University of Edinburgh（爱丁堡大学）
In recent years, there has been an increasing interest in image-based plant phenotyping, applying state-of-the-art machine learning approaches to tackle challenging problems, such as leaf segmentation (a multi-instance problem) and counting. Most of these algorithms need labelled data to learn a model for the task at hand. Despite the recent release of a few plant phenotyping datasets, large annotated plant image datasets for the purpose of training deep learning algorithms are lacking. One common approach to alleviate the lack of training data is dataset augmentation. Herein, we propose an al…
Deep learning techniques have been successfully deployed for automating plant stress identification and quantification. In recent years, there is a growing push towards training models that are interpretable -i.e. that justify their classification decisions by visually highlighting image features that were crucial for classification decisions. The expectation is that trained network models utilize image features that mimic visual cues used by plant pathologists. In this work, we compare some of the most popular interpretability methods: Saliency Maps, SmoothGrad, Guided Backpropogation, Deep T…
A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyond
Department of Electrical and Computer Engineering, Michigan State University（电气与计算机工程系，密歇根州立大学）；Department of Systems and Information Engineering & Link Lab, University of Virginia（系统与信息工程系及Link实验室，弗吉尼亚大学）；Department of Statistics and Probability, Michigan State University（统计与概率系，密歇根州立大学）；School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University（化学、化工与生物技术学院，南洋理工大学）；Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密西西比州立大学）；Department of Mechanical Engineering, Michigan State University（机械工程系，密歇根州立大学）
LeafTrackNet: A Deep Learning Framework for Robust Leaf Tracking in Top-Down Plant Phenotyping
Leibniz Institute for Agricultural Engineering（莱布尼茨农业工程与生物经济研究所）；BASF Belgium Coordination Center CommV（巴斯夫比利时协调中心）；BASF Metabolome Solutions GmbH（巴斯夫代谢组解决方案公司）；Technical University Berlin（柏林技术大学）；University of Potsdam（波茨坦大学）
Object-Centric 3D Gaussian Splatting for Strawberry Plant Reconstruction and Phenotyping
Department of Electrical and Computer Engineering, Michigan State University, East Lansing, MI, USA（电气与计算机工程系，密歇根州立大学）；Department of Mechanical Engineering, Michigan State University, East Lansing, MI, USA（机械工程系，密歇根州立大学）；Truck Crops Branch Experiment Station, Mississippi State University, Starkville, MS, USA（卡车作物分支实验站，密苏里州立大学）；Department of Agricultural and Biological Engineering, Mississippi State University, Starkville, MS, USA（农业与生物工程系，密苏里州立大学）；Tandon School of Engineering, New York University, NY, USA（坦顿工程学院，纽约大学）
China Agricultural University（中国农业大学）；University of California, Berkeley（加州大学伯克利分校）；LuDong University（鲁东大学）
Leaf segmentation is the most direct and effective way for high-throughput plant phenotype data analysis and quantitative researches of complex traits. Currently, the primary goal of plant phenotyping is to raise the accuracy of the autonomous phenotypic measurement. In this work, we present the LeafMask neural network, a new end-to-end model to delineate each leaf region and count the number of leaves, with two main components: 1) the mask assembly module merging position-sensitive bases of each predicted box after non-maximum suppression (NMS) and corresponding coefficients to generate origi…
Iowa State University（爱荷华州立大学）；Plant Science Institute, Iowa State University（爱荷华州立大学植物科学研究所）；Translational AI Research and Education Center, Iowa State University（爱荷华州立大学转化人工智能研究与教育中心）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）
United States Department of Agriculture, Agricultural Research Service（美国农业部农业研究服务局）；Appalachian Fruit Research Station (USDA-ARS-AFRS)（阿巴拉契亚水果研究站（美国农业部农业研究服务局阿巴拉契亚水果研究站））；Universidad Tecnológica de Pereira（佩雷拉理工大学）；San Joaquin Valley Agricultural Sciences Center (USDA-ARS-SJVASC)（圣华金河谷农业科学中心（美国农业部农业研究服务局圣华金河谷农业科学中心））
Images are used frequently in plant phenotyping to capture measurements. This chapter offers a repeatable method for capturing two-dimensional measurements of plant parts in field or laboratory settings using a variety of camera styles (cellular phone, DSLR), with the addition of a printed calibration pattern. The method is based on calibrating the camera using information available from the EXIF tags from the image, as well as visual information from the pattern. Code is provided to implement the method, as well as a dataset for testing. We include steps to verify protocol correctness by imag…
From UAV Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping
University of Memphis（孟菲斯大学）；University of Missouri（密苏里大学）；University of Arkansas Division of Agriculture（阿肯色大学农业分部）；Mississippi State University（密西西比州立大学）
Generating high-quality 3DMPCs by adaptive data acquisition and NeREF-based radiometric calibration with UGV plant phenotyping system
College of Biosystems Engineering and Food Science, Zhejiang University（浙江大学生物系统工程与食品科学学院）；Key Laboratory of Spectroscopy Sensing, Ministry of Agriculture and Rural Affairs（农业农村部光谱传感重点实验室）
Indian Institute of Technology Bombay（孟买印度理工学院）；ICAR-Indian Agricultural Research Institute（ICAR-印度农业研究所）
Development of either drought-resistant or drought-tolerant varieties in rice (Oryza sativa L.), especially for high yield in the context of climate change, is a crucial task across the world. The need for high yielding rice varieties is a prime concern for developing nations like India, China, and other Asian-African countries where rice is a primary staple food. The present investigation is carried out for discriminating drought tolerant, and susceptible genotypes. A total of 150 genotypes were grown under controlled conditions to evaluate at High Throughput Plant Phenomics facility, Nanaji …
ChronoRoot 2.0: An Open AI-Powered Platform for 2D Temporal Plant Phenotyping
Instituto de Fisiología, Biología Molecular y Neurociencias (IFIBYNE), CONICET-Universidad de Buenos Aires（生理学、生物分子与神经科学研究所（IFIBYNE），CONICET-布宜诺斯艾利斯大学）；APOLO Biotech（APOLO生物科技）；Instituto de Investigación en Señales, Sistemas e Inteligencia Computacional sinc(i)（信号、系统与计算智能研究所）；CONICET-Universidad Nacional del Litoral（CONICET-沿海大学）；Université Paris-Saclay, CNRS, INRAE, Université Evry, Institute of Plant Sciences Paris-Saclay (IPS2)（巴黎萨克雷大学，CNRS，INRAE，埃夫里大学，巴黎萨克雷植物科学研究所）；Université Paris Cité, CNRS, INRAE, Institute of Plant Sciences Paris-Saclay (IPS2)（巴黎城市大学，CNRS，INRAE，巴黎萨克雷植物科学研究所）
In many agricultural applications one wants to characterize physical properties of plants and use the measurements to predict, for example biomass and environmental influence. This process is known as phenotyping. Traditional collection of phenotypic information is labor-intensive and time-consuming. Use of imagery is becoming popular for phenotyping. In this paper, we present methods to estimate traits of sorghum plants from RBG cameras on board of an unmanned aerial vehicle (UAV). The position and orientation of the imagery together with the coordinates of sparse points along the area of int…
LeafInst - Unified Instance Segmentation Network for Fine-Grained Forestry Leaf Phenotype Analysis: A New UAV based Benchmark
organization= College of Information Science；Technology \& Artificial Intelligence, Nanjing Forestry University , city= Nanjing , postcode= 210037 , state= Jiangsu , country= China；organization= Department of Geography, College of Natural Resources；organization= Co-Innovation Center for Sustainable Forestry in Southern China, Nanjing Forestry University , city= Nanjing , postcode= 210037 , state= Jiangsu , country= China；organization= School of Architecture \& Urban Planning, Shenzhen University , city= Shenzhen , postcode= 518060 , state= Guangdong , country= China；organization= School of Software, Hefei University of Technology , city= Hefei , postcode= 230601 , state= Anhui , country= China
Rutgers University（罗格斯大学）；Tencent Hippocrates Research Labs（腾讯希波克拉底研究实验室）；Lieber Institute for Brain Development（利伯大脑发育研究所）
Instance segmentation of biological images is essential for studying object behaviors and properties. The challenges, such as clustering, occlusion, and adhesion problems of the objects, make instance segmentation a non-trivial task. Current box-free instance segmentation methods typically rely on local pixel-level information. Due to a lack of global object view, these methods are prone to over- or under-segmentation. On the contrary, the box-based instance segmentation methods incorporate object detection into the segmentation, performing better in identifying the individual instances. In th…
Text-conditioned Segmentation for Tomato Phenotyping via Procedural Synthetic Data
Computer, Electrical and Mathematical Sciences and Engineering Division, KAUST（计算机、电气与数学科学与工程系，沙特阿卜杜拉国王科技大学）；Department of Computer Science, University of Calgary（卡尔加里大学计算机科学系）；Department of Computer Science, Kiel University（基尔大学计算机科学系）；Department of Artificial Intelligence, Adam Mickiewicz University（亚当·密茨凯维奇大学人工智能系）；Department of Electronics and Telecommunications, Polytechnic University of Turin（都灵理工大学电子与电信系）
