Semantic segmentation of forest stands using deep learning
Faculty of Environmental Sciences and Natural Resource Management, Norwegian University of Life Sciences（生命科学大学环境科学与自然资源管理学院）；Faculty of Science and Technology, Norwegian University of Life Sciences（生命科学大学科学与技术学院）
Håkon Næss Sandum, Hans Ole Ørka, Oliver Tomic, Erik Næsset, Terje Gobakken
Forest stands are the fundamental units in forest management inventories, silviculture, and financial analysis within operational forestry. Over the past two decades, a common method for mapping stand borders has involved delineation through manual interpretation of stereographic aerial images. This is a time-consuming and subjective process, limiting operational efficiency and introducing inconsistencies. Substantial effort has been devoted to automating the process, using various algorithms together with aerial images and canopy height models constructed from airborne laser scanning (ALS) data, but manual interpretation remains the preferred method. Deep learning (DL) methods have demonstrated great potential in computer vision, yet their application to forest stand delineation remains unexplored in published research. This study presents a novel approach, framing stand delineation as a multiclass segmentation problem and applying a U-Net based DL framework. The model was trained and evaluated using multispectral images, ALS data, and an existing stand map created by an expert interpreter. Performance was assessed on independent data using overall accuracy, a standard metric for classification tasks that measures the proportions of correctly classified pixels. The model achieved an overall accuracy of 0.73. These results demonstrate strong potential for DL in automated stand delineation. However, a few key challenges were noted, especially for complex forest environments.
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）；Manipal Institute of Technology Bengaluru（马尼帕尔理工学院（班加罗尔））
Forest plays a vital role in reducing greenhouse gas emissions and mitigating climate change besides maintaining the world's biodiversity. The existing satellite-based forest monitoring system utilizes supervised learning approaches that are limited to a particular region and depend on manually annotated data to identify forest. This work envisages forest identification as a few-shot semantic segmentation task to achieve generalization across different geographical regions. The proposed few-shot segmentation approach incorporates a texture attention module in the prototypical network to highli…
Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）；School of Computer and Control Engineering, Yantai University（烟台大学计算机与控制工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
A novel multi-atlas based image segmentation method is proposed by integrating a semi-supervised label propagation method and a supervised random forests method in a pattern recognition based label fusion framework. The semi-supervised label propagation method takes into consideration local and global image appearance of images to be segmented and segments the images by propagating reliable segmentation results obtained by the supervised random forests method. Particularly, the random forests method is used to train a regression model based on image patches of atlas images for each voxel of th…
In this work, we present a random forest framework that learns the weights, shapes, and sparsities of feature representations for real-time semantic segmentation. Typical filters (kernels) have predetermined shapes and sparsities and learn only weights. A few feature extraction methods fix weights and learn only shapes and sparsities. These predetermined constraints restrict learning and extracting optimal features. To overcome this limitation, we propose an unconstrained representation that is able to extract optimal features by learning weights, shapes, and sparsities. We, then, present the …
Point2Tree(P2T) -- framework for parameter tuning of semantic and instance segmentation used with mobile laser scanning data in coniferous forest
Norwegian Institute for Bioeconomy Research (NIBIO)（挪威生物经济研究所（NIBIO））；University College London（伦敦大学学院）；NERC National Centre for Earth Observation（NERC国家地球观测中心）
University of Campinas（坎皮纳斯大学）；Jadavpur University（贾达普大学）；University of São Paulo（圣保罗大学）
Superpixel segmentation has become an important research problem in image processing. In this paper, we propose an Iterative Spanning Forest (ISF) framework, based on sequences of Image Foresting Transforms, where one can choose i) a seed sampling strategy, ii) a connectivity function, iii) an adjacency relation, and iv) a seed pixel recomputation procedure to generate improved sets of connected superpixels (supervoxels in 3D) per iteration. The superpixels in ISF structurally correspond to spanning trees rooted at those seeds. We present five ISF methods to illustrate different choices of its…
Label-Efficient 3D Forest Mapping: Self-Supervised and Transfer Learning for Instance Segmentation, Semantic Segmentation, and Species Classification
Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Helmholtz Institute Freiberg for Resource Technology (HIF)（德累斯顿-罗斯托克亥姆霍兹中心（HZDR）、弗里贝格资源技术亥姆霍兹研究所（HIF））；Remote Sensing and Geoinformatics, Freie Universität Berlin（柏林自由大学遥感与地理信息学系）；Institute of Geomatics, BOKU University（博科尼大学测绘学院）；Faculty of Electrical and Computer Engineering, University of Iceland（爱沙尼亚大学电气与计算机工程学院）
Universidade Federal de Mato Grosso do Sul（南马托格罗索联邦大学）
The task of counting eucalyptus trees from aerial images collected by unmanned aerial vehicles (UAVs) has been frequently explored by techniques of estimation of the basal area, i.e, by determining the expected number of trees based on sampling techniques. An alternative is the use of machine learning to identify patterns that represent a tree unit, and then search for the occurrence of these patterns throughout the image. This strategy depends on a supervised image segmentation step to define predefined interest regions. Thus, it is possible to automate the counting of eucalyptus trees in the…
Imperial College London（帝国理工学院）；Royal Brompton Hospital（皇家布朗普顿医院）
Myocardial contrast echocardiography (MCE) is an imaging technique that assesses left ventricle function and myocardial perfusion for the detection of coronary artery diseases. Automatic MCE perfusion quantification is challenging and requires accurate segmentation of the myocardium from noisy and time-varying images. Random forests (RF) have been successfully applied to many medical image segmentation tasks. However, the pixel-wise RF classifier ignores contextual relationships between label outputs of individual pixels. RF which only utilizes local appearance features is also susceptible to …
SelectAnyTree: A Promptable Instance Segmentation Model for 3D Forest LiDAR Point Clouds
Nagoya University（名古屋大学）；RIKEN（理化学研究所）；University of Freiburg（弗莱堡大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Twente（特温特大学）；KC Machine Learning Lab（KC机器学习实验室）；Hanoi University of Science and Technology（河内科技大学）；Ritsumeikan University（立命馆大学）
Global environment monitoring is a task that requires additional attention in the contemporary rapid climate change environment. This includes monitoring the rate of deforestation and areas affected by flooding. Satellite imaging has greatly helped monitor the earth, and deep learning techniques have helped to automate this monitoring process. This paper proposes a solution for observing the area covered by the forest and water. To achieve this task UNet model has been proposed, which is an image segmentation model. The model achieved a validation accuracy of 82.55% and 82.92% for the segmenta…
University of Campinas (UNICAMP)（坎皮纳斯大学）；Pontificial Catholic University of Minas Gerais (PUC-Minas)（米纳斯吉拉斯天主教大学）
As constituent parts of image objects, superpixels can improve several higher-level operations. However, image segmentation methods might have their accuracy seriously compromised for reduced numbers of superpixels. We have investigated a solution based on the Iterative Spanning Forest (ISF) framework. In this work, we present Dynamic ISF (DISF) -- a method based on the following steps. (a) It starts from an image graph and a seed set with considerably more pixels than the desired number of superpixels. (b) The seeds compete among themselves, and each seed conquers its most closely connected p…
Heidelberg University（海德堡大学）；Harvard Medical School（哈佛医学院）；BIH/MDC（柏林卫生研究院/马克斯·德尔布吕克分子医学中心）；University of Twente（特文特大学）；MPI-CBG（马克斯·普朗克分子细胞生物学与遗传学研究所）
We consider the task of pixel-wise semantic segmentation given a small set of labeled training images. Among two of the most popular techniques to address this task are Decision Forests (DF) and Neural Networks (NN). In this work, we explore the relationship between two special forms of these techniques: stacked DFs (namely Auto-context) and deep Convolutional Neural Networks (ConvNet). Our main contribution is to show that Auto-context can be mapped to a deep ConvNet with novel architecture, and thereby trained end-to-end. This mapping can be used as an initialization of a deep ConvNet, enabl…
State University of New York at Buffalo（纽约州立大学布法罗分校）；Tianjin University（天津大学）；Horizon Robotics, Inc.（地平线机器人科技有限公司）
Despite the previous success of object analysis, detecting and segmenting a large number of object categories with a long-tailed data distribution remains a challenging problem and is less investigated. For a large-vocabulary classifier, the chance of obtaining noisy logits is much higher, which can easily lead to a wrong recognition. In this paper, we exploit prior knowledge of the relations among object categories to cluster fine-grained classes into coarser parent classes, and construct a classification tree that is responsible for parsing an object instance into a fine-grained category via…
3D forest semantic segmentation using multispectral LiDAR and 3D deep learning
Bruno Kessler Foundation (FBK)（布鲁诺·凯塞勒基金会）；TU Wien（维也纳工业大学）；Finnish Geospatial Research Institute（芬兰地理空间研究所）；National Land Survey of Finland（芬兰国家土地测量局）
LeafInst - Unified Instance Segmentation Network for Fine-Grained Forestry Leaf Phenotype Analysis: A New UAV based Benchmark
organization= College of Information Science；Technology \& Artificial Intelligence, Nanjing Forestry University , city= Nanjing , postcode= 210037 , state= Jiangsu , country= China；organization= Department of Geography, College of Natural Resources；organization= Co-Innovation Center for Sustainable Forestry in Southern China, Nanjing Forestry University , city= Nanjing , postcode= 210037 , state= Jiangsu , country= China；organization= School of Architecture \& Urban Planning, Shenzhen University , city= Shenzhen , postcode= 518060 , state= Guangdong , country= China；organization= School of Software, Hefei University of Technology , city= Hefei , postcode= 230601 , state= Anhui , country= China
University of Notre Dame（圣母大学）；Nanjing University of Science and Technology（南京理工大学）；Leibniz-Institut für Analytische Wissenschaften–ISAS–e.V.（莱布尼茨分析科学研究所）
Deep learning (DL) based semantic segmentation methods have achieved excellent performance in biomedical image segmentation, producing high quality probability maps to allow extraction of rich instance information to facilitate good instance segmentation. While numerous efforts were put into developing new DL semantic segmentation models, less attention was paid to a key issue of how to effectively explore their probability maps to attain the best possible instance segmentation. We observe that probability maps by DL semantic segmentation models can be used to generate many possible instance c…
Remote Sensing Technology Institute, German Aerospace Center（德国航空航天中心遥感技术研究所）；Technical University of Munich（慕尼黑工业大学）；TJ Watson Research Center, IBM Research（IBM研究院托马斯·J·沃森研究中心）
We present and evaluate a weakly-supervised methodology to quantify the spatio-temporal distribution of urban forests based on remotely sensed data with close-to-zero human interaction. Successfully training machine learning models for semantic segmentation typically depends on the availability of high-quality labels. We evaluate the benefit of high-resolution, three-dimensional point cloud data (LiDAR) as source of noisy labels in order to train models for the localization of trees in orthophotos. As proof of concept we sense Hurricane Sandy's impact on urban forests in Coney Island, New York…
Highly Detailed and Generalizable Broadleaf Tree Crown Instance Segmentation from UAV Imagery
DeepForest Technologies Co., Ltd.（深森林技术有限公司）；YM Lab.（YM实验室）；Graduate School of Agriculture, Kyoto University（京都大学农业研究院）；Graduate School of Science, Osaka Metropolitan University（大阪 metropolitan 大学理学研究院）；Faculty of Tropical Forestry, Universiti Malaysia Sabah（马来西亚沙巴大学热带林业学院）；Forest Research Centre, Sabah Forestry Department（沙巴林业部门森林研究中心）
Accurate semantic labeling of image pixels is difficult because intra-class variability is often greater than inter-class variability. In turn, fast semantic segmentation is hard because accurate models are usually too complicated to also run quickly at test-time. Our experience with building and running semantic segmentation systems has also shown a reasonably obvious bottleneck on model complexity, imposed by small training datasets. We therefore propose two simple complementary strategies that leverage context to give better semantic segmentation, while scaling up or down to train on differ…
In this paper, we propose a method for the automatic semantic segmentation of satellite images into six classes (sparse forest, dense forest, moor, herbaceous formation, building, and road). We rely on Swin Transformer architecture and build the dataset from IGN open data. We report quantitative and qualitative segmentation results on this dataset and discuss strengths and limitations. The dataset and the trained model are made publicly available.
This survey gives an overview over different techniques used for pixel-level semantic segmentation. Metrics and datasets for the evaluation of segmentation algorithms and traditional approaches for segmentation such as unsupervised methods, Decision Forests and SVMs are described and pointers to the relevant papers are given. Recently published approaches with convolutional neural networks are mentioned and typical problematic situations for segmentation algorithms are examined. A taxonomy of segmentation algorithms is given.
Edge detection is a critical component of many vision systems, including object detectors and image segmentation algorithms. Patches of edges exhibit well-known forms of local structure, such as straight lines or T-junctions. In this paper we take advantage of the structure present in local image patches to learn both an accurate and computationally efficient edge detector. We formulate the problem of predicting local edge masks in a structured learning framework applied to random decision forests. Our novel approach to learning decision trees robustly maps the structured labels to a discrete …
Pontifical Catholic University of Rio de Janeiro（里约热内卢天主教大学）；IBM Research（IBM研究院）；McMaster University（麦克马斯特大学）；National Institute for Space Research (INPE)（巴西国家空间研究所）；Santa Catarina State University (UDESC)（圣卡塔琳娜州立大学）
This work proposes a multi-task fully convolutional architecture for tree species mapping in dense forests from sparse and scarce polygon-level annotations using hyperspectral UAV-borne data. Our model implements a partial loss function that enables dense tree semantic labeling outcomes from non-dense training samples, and a distance regression complementary task that enforces tree crown boundary constraints and substantially improves the model performance. Our multi-task architecture uses a shared backbone network that learns common representations for both tasks and two task-specific decoder…
Planetary exploration depends heavily on 3D image data to characterize the static and dynamic properties of the rock and environment. Analyzing 3D images requires many computations, causing efficiency to suffer lengthy processing time alongside large energy consumption. High-Performance Computing (HPC) provides apparent efficiency at the expense of energy consumption. However, for remote explorations, the conveyed surveillance and the robotized sensing need faster data analysis with ultimate accuracy to make real-time decisions. In such environments, access to HPC and energy is limited. Theref…
RWTH Aachen University（亚琛工业大学）；University of Twente（特温特大学）；Monash University（莫纳什大学）；AICES Graduate School（AICES研究生院）
The hemocompatibility of blood-contacting medical devices remains one of the major challenges in biomedical engineering and makes research in the field of new and improved materials inevitable. However, current in-vitro test and analysis methods are still lacking standardization and comparability, which impedes advances in material design. For example, the optical platelet analysis of material in-vitro hemocompatibility tests is carried out manually or semi-manually by each research group individually. As a step towards standardization, this paper proposes an automation approach for the optica…
Unsupervised deep learning for semantic segmentation of multispectral LiDAR forest point clouds
Department of Remote Sensing and Photogrammetry, Finnish Geospatial Research Institute FGI, The National Land Survey of Finland（遥感与摄影测量系，芬兰地理研究 institute FGI，芬兰国家土地测绘局）；D Optical Metrology (3DOM) unit, Bruno Kessler Foundation (FBK)（3D光学计量（3DOM）单元，布鲁诺·凯瑟尔基金会（FBK））；Department of Built Environment, Aalto University, School of Engineering（环境学院，艾尔托大学，工程学院）
Vision-based segmentation in forested environments is a key functionality for autonomous forestry operations such as tree felling and forwarding. Deep learning algorithms demonstrate promising results to perform visual tasks such as object detection. However, the supervised learning process of these algorithms requires annotations from a large diversity of images. In this work, we propose to use simulated forest environments to automatically generate 43 k realistic synthetic images with pixel-level annotations, and use it to train deep learning algorithms for tree detection. This allows us to …
Wood logs picking is a challenging task to automate. Indeed, logs usually come in cluttered configurations, randomly orientated and overlapping. Recent work on log picking automation usually assume that the logs' pose is known, with little consideration given to the actual perception problem. In this paper, we squarely address the latter, using a data-driven approach. First, we introduce a novel dataset, named TimberSeg 1.0, that is densely annotated, i.e., that includes both bounding boxes and pixel-level mask annotations for logs. This dataset comprises 220 images with 2500 individually segm…
SilvaScenes: Tree Detection and Species Classification from Under-Canopy Images in Natural Forests
Northern Robotics Laboratory, Université Laval（北方机器人实验室，拉瓦尔大学）；Département des sciences du bois et de la forêt, Université Laval（林业与木材科学系，拉瓦尔大学）；Institut des Sciences de la Forêt tempérée, Université du Québec en Outaouais（温带森林科学研究所，魁北克outsouais大学）
We introduce a new machine learning approach for image segmentation that uses a neural network to model the conditional energy of a segmentation given an image. Our approach, combinatorial energy learning for image segmentation (CELIS) places a particular emphasis on modeling the inherent combinatorial nature of dense image segmentation problems. We propose efficient algorithms for learning deep neural networks to model the energy function, and for local optimization of this energy in the space of supervoxel agglomerations. We extensively evaluate our method on a publicly available 3-D microsc…
In this paper we present Latent-Class Hough Forests, a method for object detection and 6 DoF pose estimation in heavily cluttered and occluded scenarios. We adapt a state of the art template matching feature into a scale-invariant patch descriptor and integrate it into a regression forest using a novel template-based split function. We train with positive samples only and we treat class distributions at the leaf nodes as latent variables. During testing we infer by iteratively updating these distributions, providing accurate estimation of background clutter and foreground occlusions and, thus,…
treeX: Unsupervised Tree Instance Segmentation in Dense Forest Point Clouds
University of Potsdam（波恩大学）；Digital Engineering Faculty（数字工程学院）；BOKU University（博科尼大学）；Department of Ecosystem Management, Climate and Biodiversity（生态系统管理、气候与生物多样性部门）；Institute of Forest Growth（森林生长研究所）；Eberswalde University of Sustainable Development（埃伯斯瓦尔德可持续发展大学）；State Forestry Research Centre（州林业研究机构）；Hasso Plattner Institute for Digital Engineering（哈索·plattnner数字工程研究所）
The Hong Kong Polytechnic University（香港理工大学）；Bavarian Forest National Park（巴伐利亚森林国家公园）
In this paper, we introduce a framework for segmenting instances of a common object class by multiple active contour evolution over semantic segmentation maps of images obtained through fully convolutional networks. The contour evolution is cast as an energy minimization problem, where the aggregate energy functional incorporates a data fit term, an explicit shape model, and accounts for object overlap. Efficient solution neighborhood operators are proposed, enabling optimization through metaheuristics such as simulated annealing. We instantiate the proposed framework in the context of segment…
University of Pennsylvania（宾夕法尼亚大学）；University of Sao Paulo（圣保罗大学）
This paper describes an end-to-end pipeline for tree diameter estimation based on semantic segmentation and lidar odometry and mapping. Accurate mapping of this type of environment is challenging since the ground and the trees are surrounded by leaves, thorns and vines, and the sensor typically experiences extreme motion. We propose a semantic feature based pose optimization that simultaneously refines the tree models while estimating the robot pose. The pipeline utilizes a custom virtual reality tool for labeling 3D scans that is used to train a semantic segmentation network. The masked point…
Concordia Institute for Information Systems Engineering（康考迪亚信息系统工程研究所）；Concordia University（康考迪亚大学）；Université TELUQ（TELUQ大学）
Site preparation by mounding is a commonly used silvicultural treatment that improves tree growth conditions by mechanically creating planting microsites called mounds. Following site preparation, the next critical step is to count the number of mounds, which provides forest managers with a precise estimate of the number of seedlings required for a given plantation block. Counting the number of mounds is generally conducted through manual field surveys by forestry workers, which is costly and prone to errors, especially for large areas. To address this issue, we present a novel framework explo…
Mandible bone segmentation from computed tomography (CT) scans is challenging due to mandible's structural irregularities, complex shape patterns, and lack of contrast in joints. Furthermore, connections of teeth to mandible and mandible to remaining parts of the skull make it extremely difficult to identify mandible boundary automatically. This study addresses these challenges by proposing a novel framework where we define the segmentation as two complementary tasks: recognition and delineation. For recognition, we use random forest regression to localize mandible in 3D. For delineation, we p…
Institute of Neuroinformatics（神经信息学研究所）；University of Zurich（苏黎世大学）；ETH Zurich（苏黎世联邦理工学院）
We present a method for segmenting neuron membranes in 2D electron microscopy imagery. This segmentation task has been a bottleneck to reconstruction efforts of the brain's synaptic circuits. One common problem is the misclassification of blurry membrane fragments as cell interior, which leads to merging of two adjacent neuron sections into one via the blurry membrane region. Human annotators can easily avoid such errors by implicitly performing gap completion, taking into account the continuity of membranes. Drawing inspiration from these human strategies, we formulate the segmentation task a…
The collection of a high number of pixel-based labeled training samples for tree species identification is time consuming and costly in operational forestry applications. To address this problem, in this paper we investigate the effectiveness of explanation methods for deep neural networks in performing weakly supervised semantic segmentation using only image-level labels. Specifically, we consider four methods:i) class activation maps (CAM); ii) gradient-based CAM; iii) pixel correlation module; and iv) self-enhancing maps (SEM). We compare these methods with each other using both quantitativ…
Institute of Computing, University of Campinas（坎皮纳斯大学计算研究所）
The semantic segmentation (SS) task aims to create a dense classification by labeling at the pixel level each object present on images. Convolutional neural network (CNN) approaches have been widely used, and exhibited the best results in this task. However, the loss of spatial precision on the results is a main drawback that has not been solved. In this work, we propose to use a multi-task approach by complementing the semantic segmentation task with edge detection, semantic contour, and distance transform tasks. We propose that by sharing a common latent space, the complementary tasks can pr…
CentraleSupelec（巴黎中央高等电力学院）；Inria（法国国家信息与自动化研究所）；Universite Paris-Saclay（巴黎萨克雷大学）；Polytechnique Montreal（蒙特利尔理工学院）；University of Montreal（蒙特利尔大学）；Sainte-Justine Hospital Research Center（圣朱斯汀医院研究中心）
In this paper we propose a deep learning approach for segmenting sub-cortical structures of the human brain in Magnetic Resonance (MR) image data. We draw inspiration from a state-of-the-art Fully-Convolutional Neural Network (F-CNN) architecture for semantic segmentation of objects in natural images, and adapt it to our task. Unlike previous CNN-based methods that operate on image patches, our model is applied on a full blown 2D image, without any alignment or registration steps at testing time. We further improve segmentation results by interpreting the CNN output as potentials of a Markov R…
Yokohama City University Medical Center（横滨市立大学医学中心）
We present a fully automatic method employing convolutional neural networks based on the 2D U-net architecture and random forest classifier to solve the automatic liver lesion segmentation problem of the ISBI 2017 Liver Tumor Segmentation Challenge (LiTS). In order to constrain the ROI in which the tumors could be located, a liver segmentation is performed first. For the organ segmentation, an ensemble of convolutional networks is trained to segment a liver using a set of 179 liver CT datasets from liver surgery planning. Inside of the liver ROI a neural network, trained using 127 challenge tr…
In this paper, we propose an efficient semantic segmentation framework for indoor scenes, tailored to the application on a mobile robot. Semantic segmentation can help robots to gain a reasonable understanding of their environment, but to reach this goal, the algorithms not only need to be accurate, but also fast and robust. Therefore, we developed an optimized 3D point cloud processing framework based on a Randomized Decision Forest, achieving competitive results at sufficiently high frame rates. We evaluate the capabilities of our method on the popular NYU depth dataset and our own data and …
