Tree-GPT: Modular Large Language Model Expert System for Forest Remote Sensing Image Understanding and Interactive Analysis
Shenzhen University（深圳大学）；Research Institute for Smart Cities（智慧城市研究院）；State Key Laboratory of Subtropical Building and Urban Science（亚热带建筑与城市科学国家重点实验室）；Guangdong–Hong Kong-Macau Joint Laboratory for Smart Cities（粤港澳智慧城市联合实验室）
Siqi Du, Shengjun Tang, Weixi Wang, Xiaoming Li, Renzhong Guo
本文提出了一种名为Tree-GPT的新型框架，将大语言模型（Large Language Models，LLMs）融入林业遥感数据工作流，从而提升数据分析效率。当前，LLMs无法从图像中提取或理解信息，且可能因缺乏领域知识生成不准确的文本，这限制了其在林业数据分析中的应用。为解决该问题，我们提出了模块化的LLM专家系统Tree-GPT，它集成了图像理解模块、领域知识库与工具链，使LLMs具备图像理解、获取准确知识、生成代码并在本地环境中执行数据分析的能力。具体而言，图像理解模块通过自动或交互式生成提示词，引导Segment Anything Model（SAM）生成并选择最优的树木分割结果，以此从森林遥感图像中提取结构化信息；系统随后基于这些结果计算树木结构参数，并将其存储在数据库中。接收到特定自然语言指令后，LLM会基于思维链生成代码以完成分析任务，代码随后由LLM智能体在本地环境中执行。针对生态参数计算，系统会从知识库中检索对应知识并输入LLM，以引导其生成准确的代码。我们在搜索、可视化、机器学习分析等多项任务上测试了该系统，原型系统表现良好，展现了LLMs在林业研究与环境科学中动态应用的潜力。
英文摘要：
This paper introduces a novel framework, Tree-GPT, which incorporates Large Language Models (LLMs) into the forestry remote sensing data workflow, thereby enhancing the efficiency of data analysis. Currently, LLMs are unable to extract or comprehend information from images and may generate inaccurate text due to a lack of domain knowledge, limiting their use in forestry data analysis. To address this issue, we propose a modular LLM expert system, Tree-GPT, that integrates image understanding modules, domain knowledge bases, and toolchains. This empowers LLMs with the ability to comprehend images, acquire accurate knowledge, generate code, and perform data analysis in a local environment. Specifically, the image understanding module extracts structured information from forest remote sensing images by utilizing automatic or interactive generation of prompts to guide the Segment Anything Model (SAM) in generating and selecting optimal tree segmentation results. The system then calculates tree structural parameters based on these results and stores them in a database. Upon receiving a specific natural language instruction, the LLM generates code based on a thought chain to accomplish the analysis task. The code is then executed by an LLM agent in a local environment and . For ecological parameter calculations, the system retrieves the corresponding knowledge from the knowledge base and inputs it into the LLM to guide the generation of accurate code. We tested this system on several tasks, including Search, Visualization, and Machine Learning Analysis. The prototype system performed well, demonstrating the potential for dynamic usage of LLMs in forestry research and environmental sciences.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）；Ulm University（乌尔姆大学）
For many countries like Russia, Canada, or the USA, a robust and detailed tree species inventory is essential to manage their forests sustainably. Since one can not apply unmanned aerial vehicle (UAV) imagery-based approaches to large-scale forest inventory applications, the utilization of machine learning algorithms on satellite imagery is a rising topic of research. Although satellite imagery quality is relatively low, additional spectral channels provide a sufficient amount of information for tree crown classification tasks. Assuming that tree crowns are detected already, we use embeddings …
FoMo: Multi-Modal, Multi-Scale and Multi-Task Remote Sensing Foundation Models for Forest Monitoring
National Observatory of Athens（雅典国家天文台）；National Technical University of Athens（雅典国家技术大学）；Harokopio University of Athens（哈罗科皮奥大学）；McGill University（麦吉尔大学）
Over the last few decades, deforestation and climate change have caused increasing number of forest fires. In Southeast Asia, Indonesia has been the most affected country by tropical peatland forest fires. These fires have a significant impact on the climate resulting in extensive health, social and economic issues. Existing forest fire prediction systems, such as the Canadian Forest Fire Danger Rating System, are based on handcrafted features and require installation and maintenance of expensive instruments on the ground, which can be a challenge for developing countries such as Indonesia. We…
UiT The Arctic University of Norway（挪威北极大学）；NORCE Norwegian Research Centre（挪威NORCE研究中心）；Norwegian Institute for Nature Research（挪威自然研究所）
Several generic methods have recently been developed for change detection in heterogeneous remote sensing data, such as images from synthetic aperture radar (SAR) and multispectral radiometers. However, these are not well suited to detect weak signatures of certain disturbances of ecological systems. To resolve this problem we propose a new approach based on image-to-image translation and one-class classification (OCC). We aim to map forest mortality caused by an outbreak of geometrid moths in a sparsely forested forest-tundra ecotone using multisource satellite images. The images preceding an…
Modeling forests using historical data allows for more accurately evolution analysis, thus providing an important basis for other studies. As a recognized and effective tool, remote sensing plays an important role in forestry analysis. We can use it to derive information about the forest, including tree type, coverage and canopy density. There are many forest time series modeling studies using statistic values, but few using remote sensing images. Image prediction digital twin is an implementation of digital twin, which aims to predict future images bases on historical data. In this paper, we …
Estimating forest carbon stocks from high-resolution remote sensing imagery by reducing domain shift with style transfer
School of Geography and Remote Sensing, Guangzhou University（广州大学地理与遥感学院）；Innovation Center for Remote Sensing Big Data Intelligent Applications, Guangzhou University（广州大学遥感大数据智能应用创新中心）
UiT The Arctic University of Norway（挪威北极大学）；University of Genoa（热那亚大学）
Change detection in heterogeneous multitemporal satellite images is an emerging topic in remote sensing. In this paper we propose a framework, based on image regression, to perform change detection in heterogeneous multitemporal satellite images, which has become a main topic in remote sensing. Our method learns a transformation to map the first image to the domain of the other image, and vice versa. Four regression methods are selected to carry out the transformation: Gaussian processes, support vector machines, random forests, and a recently proposed kernel regression method called homogeneo…
Shanghai University（上海大学）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Cloud detection plays a very important role in the process of remote sensing images. This paper designs a super-pixel level cloud detection method based on convolutional neural network (CNN) and deep forest. Firstly, remote sensing images are segmented into super-pixels through the combination of SLIC and SEEDS. Structured forests is carried out to compute edge probability of each pixel, based on which super-pixels are segmented more precisely. Segmented super-pixels compose a super-pixel level remote sensing database. Though cloud detection is essentially a binary classification problem, our …
There is much current interest in using multi-sensor airborne remote sensing to monitor the structure and biodiversity of forests. This paper addresses the application of non-parametric image registration techniques to precisely align images obtained from multimodal imaging, which is critical for the successful identification of individual trees using object recognition approaches. Non-parametric image registration, in particular the technique of optimizing one objective function containing data fidelity and regularization terms, provides flexible algorithms for image registration. Using a sur…
JOANNEUM RESEARCH（约阿内姆研究公司）；Albert-Ludwigs-University Freiburg（弗赖堡阿尔伯特-路德维希大学）；University of Patras（帕特雷大学）
Purpose of review: This paper presents a review of the current state of the art in remote sensing based monitoring of forest disturbances and forest degradation from optical Earth Observation data. Part one comprises an overview of currently available optical remote sensing sensors, which can be used for forest disturbance and degradation mapping. Part two reviews the two main categories of existing approaches: classical image-to-image change detection and time series analysis. Recent findings: With the launch of the Sentinel-2a satellite and available Landsat imagery, time series analysis has…
Rainforests play an important role in the global ecosystem. However, significant regions of them are facing deforestation and degradation due to several reasons. Diverse government and private initiatives were created to monitor and alert for deforestation increases from remote sensing images, using different ways to deal with the notable amount of generated data. Citizen Science projects can also be used to reach the same goal. Citizen Science consists of scientific research involving nonprofessional volunteers for analyzing, collecting data, and using their computational resources to outcome…
Airborne discrete return light detection and ranging (LiDAR) point clouds covering forested areas can be processed to segment individual trees and retrieve their morphological attributes. Segmenting individual trees in natural deciduous forests however remained a challenge because of the complex and multi-layered canopy. In this chapter, we present (i) a robust segmentation method that avoids a priori assumptions about the canopy structure, (ii) a vertical canopy stratification procedure that improves segmentation of understory trees, (iii) an occlusion model for estimating the point density o…
Comparing remote sensing-based forest biomass mapping approaches using new forest inventory plots in contrasting forests in northeastern and southwestern China
School of GeoSciences, University of Edinburgh（爱丁堡大学地球科学学院）；Space Intelligence Ltd（空间智能有限公司）；Finnish Geospatial Research Institute（芬兰地理空间研究所）；Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Department of Forest Science, University of Helsinki（赫尔辛基大学森林科学系）；Jihua Lab（季华实验室）
Forestpest-YOLO: A High-Performance Detection Framework for Small Forestry Pests
School of Advanced Manufacturing, Guangdong University of Technology（广东技术大学先进制造学院）；Guangdong Power Grid Co.,Ltd.（广东电网公司）；School of Computer Science and Engineering, Huizhou University（惠州市大学计算机科学与工程学院）；Obstetrics and Gynaecology, Huizhou First Maternal and Child Health Care Hospital（惠州市第一人民医院妇产科）
FUSAR-GPT : A Spatiotemporal Feature-Embedded and Two-Stage Decoupled Visual Language Model for SAR Imagery
Fudan University（复旦大学）；Discipline and Technology Center of Microwave Vision Intelligent Sensing, Fudan University（微波视觉智能感知学科与技术中心，复旦大学）；Shanghai Innovation Institute（上海创新研究院）
University of Padova（帕多瓦大学）；Fondazione Edmund Mach（埃德蒙·马赫基金会）；Provincia Autonoma di Trento（特伦托自治省）
This work intends to lay the foundations for identifying the prevailing forest types and the delineation of forest units within private forest inventories in the Autonomous Province of Trento (PAT), using currently available remote sensing solutions. In particular, data from LiDAR and hyperspectral surveys of 2014 made available by PAT were acquired and processed. Such studies are very important in the context of forest management scenarios. The method includes defining tree species ground-truth by outlining single tree crowns with polygons and labeling them. Successively two supervised machin…
Airborne laser scanning (LiDAR) point clouds over large forested areas can be processed to segment individual trees and subsequently extract tree-level information. Existing segmentation procedures typically detect more than 90% of overstory trees, yet they barely detect 60% of understory trees because of the occlusion effect of higher canopy layers. Although understory trees provide limited financial value, they are an essential component of ecosystem functioning by offering habitat for numerous wildlife species and influencing stand development. Here we model the occlusion effect in terms of…
Predicting wildfire spread is critical for land management and disaster preparedness. To this end, we present `Next Day Wildfire Spread,' a curated, large-scale, multivariate data set of historical wildfires aggregating nearly a decade of remote-sensing data across the United States. In contrast to existing fire data sets based on Earth observation satellites, our data set combines 2D fire data with multiple explanatory variables (e.g., topography, vegetation, weather, drought index, population density) aligned over 2D regions, providing a feature-rich data set for machine learning. To demonst…
Training point-based deep learning networks for forest segmentation with synthetic data
Instituto en Ciencias de la Computación (UBA-CONICET)（计算机科学研究所（UBA-CONICET））；Departamento de Computación, Facultad de Ciencias Exactas y Naturales, Universidad de Buenos Aires（布宜诺斯艾利斯大学 精确与自然科学学院 计算机系）
OpenForest: A data catalogue for machine learning in forest monitoring
McGill University（麦吉尔大学）；Mila, Quebec AI Institute（魁北克人工智能研究所（Mila））；Leipzig University（莱比锡大学）；German Centre for Integrative Biodiversity Research (iDiv)（德国综合生物多样性研究中心）；Université de Montréal（蒙特利尔大学）
The collection of a high number of pixel-based labeled training samples for tree species identification is time consuming and costly in operational forestry applications. To address this problem, in this paper we investigate the effectiveness of explanation methods for deep neural networks in performing weakly supervised semantic segmentation using only image-level labels. Specifically, we consider four methods:i) class activation maps (CAM); ii) gradient-based CAM; iii) pixel correlation module; and iv) self-enhancing maps (SEM). We compare these methods with each other using both quantitativ…
Identifying regions that have high likelihood for wildfires is a key component of land and forestry management and disaster preparedness. We create a data set by aggregating nearly a decade of remote-sensing data and historical fire records to predict wildfires. This prediction problem is framed as three machine learning tasks. Results are compared and analyzed for four different deep learning models to estimate wildfire likelihood. The results demonstrate that deep learning models can successfully identify areas of high fire likelihood using aggregated data about vegetation, weather, and topo…
Classification of Single Tree Decay Stages from Combined Airborne LiDAR Data and CIR Imagery
The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）；Bavarian Forest National Park（巴伐利亚森林国家公园）；University of Freiburg（弗莱堡大学）；Inland Norway University of Applied Sciences（挪威内陆应用科学大学）
Transformers vs. Recurrent Models for Estimating Forest Gross Primary Production
IEF, Leipzig University（莱比锡大学IEF）；iDiv；MPI PKS（马克斯·普朗克研究所）；IPL, Universitat de València（瓦伦西亚大学IPL）；Bioclimatology, University of Göttingen（哥廷根大学生物气候学系）；GEMA, Universidad de Medellín（梅迪纳大学GEMA）
Finnish Environment Institute（芬兰环境研究所）；Metsähallitus, National Parks（芬兰林业管理局国家公园）
Remote sensing benefits habitat conservation by making monitoring of large areas easier compared to field surveying especially if the remote sensed data can be automatically analyzed. An important aspect of monitoring is classifying and mapping habitat types present in the monitored area. Automatic classification is a difficult task, as classes have fine-grained differences and their distributions are long-tailed and unbalanced. Usually training data used for automatic land cover classification relies on fully annotated segmentation maps, annotated from remote sensed imagery to a fairly high-l…
RSGPT: A Remote Sensing Vision Language Model and Benchmark
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；New York University Abu Dhabi（纽约大学阿布扎比分校）；University of Chinese Academy of Sciences（中国科学院大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Texture classification is a problem that has various applications such as remote sensing and forest species recognition. Solutions tend to be custom fit to the dataset used but fails to generalize. The Convolutional Neural Network (CNN) in combination with Support Vector Machine (SVM) form a robust selection between powerful invariant feature extractor and accurate classifier. The fusion of experts provides stability in classification rates among different datasets.
A method for estimating forest carbon storage distribution density via artificial intelligence generated content model
School of Geography and Remote Sensing, Guangzhou University（广州大学地理与遥感学院）；Innovation Center for Remote Sensing Big Data Intelligent Applications, Guangzhou University（广州大学遥感大数据智能应用创新中心）
Université de Moncton（蒙克顿大学）；Universidad Técnica Federico Santa María（费德里科圣玛丽亚技术大学）
Wildfires represent an important natural risk causing economic losses, human death and important environmental damage. In recent years, we witness an increase in fire intensity and frequency. Research has been conducted towards the development of dedicated solutions for wildland and forest fire assistance and fighting. Systems were proposed for the remote detection and tracking of fires. These systems have shown improvements in the area of efficient data collection and fire characterization within small scale environments. However, wildfires cover large areas making some of the proposed ground…
Tomographic SAR Reconstruction for Forest Height Estimation
University of Cambridge（剑桥大学）；United Nations Office for Outer Space Affairs（联合国外层空间事务厅）；UK Department for Energy Security and Net Zero（英国能源安全与净零部）；INSEC TEC；Drexel University（卓克索大学）
本文提出直接由二维 SLC SAR 图像利用深度学习估计森林冠层高度，并评估 SLC 数量影响；结果表明绕过完整层析处理会使误差增加 16%–21%。
Remote Sensing Technology Institute, German Aerospace Center（德国航空航天中心遥感技术研究所）；Technical University of Munich（慕尼黑工业大学）；TJ Watson Research Center, IBM Research（IBM研究院托马斯·J·沃森研究中心）
We present and evaluate a weakly-supervised methodology to quantify the spatio-temporal distribution of urban forests based on remotely sensed data with close-to-zero human interaction. Successfully training machine learning models for semantic segmentation typically depends on the availability of high-quality labels. We evaluate the benefit of high-resolution, three-dimensional point cloud data (LiDAR) as source of noisy labels in order to train models for the localization of trees in orthophotos. As proof of concept we sense Hurricane Sandy's impact on urban forests in Coney Island, New York…
Kyoto University（京都大学）；Japan Science and Technology Agency（日本科学技术振兴机构）
Automatic classification of trees using remotely sensed data has been a dream of many scientists and land use managers. Recently, Unmanned aerial vehicles (UAV) has been expected to be an easy-to-use, cost-effective tool for remote sensing of forests, and deep learning has attracted attention for its ability concerning machine vision. In this study, using a commercially available UAV and a publicly available package for deep learning, we constructed a machine vision system for the automatic classification of trees. In our method, we segmented a UAV photography image of forest into individual t…
