IC-EO: Interpretable Code-based assistant for Earth Observation
Université Paris Cité, LIPADE, F-75006 Paris, France askEarth AG, Zurich, Switzerland ESA -lab, Frascati, Italy ESA, Advanced Concepts;Studies Office, Noordwijk, the Netherlands
尽管计算机视觉近年来取得了进展,但地球观测(EO)分析仍难以让非专业人士操作,需要专业知识和技术能力。此外,许多系统返回的黑盒预测难以审计或复现。利用最近在工具LLMs方面的进展,本研究提出了一种对话式、生成代码的代理,将自然语言查询转换为可执行、可审计的Python工作流程。该代理运行于一个统一的、易于扩展的API上,用于分类、分割、检测(定向边界框)、光谱指数和地理空间运算符。通过我们提出的框架,可以在三个层面控制结果:(i)工具层面在公开的EO基准上的性能;(ii)在代理层面以理解生成有效、无幻觉代码的能力;(iii)在特定使用案例层面。在本工作中,我们选择了两个感兴趣的使用案例:土地组成制图和野火后损害评估。所提出的代理在一般用途LLM/VLM基线(GPT-4o、LLaVA)上表现更优,土地组成准确率达64.2% vs 51.7%,野火后分析准确率达50% vs 0%,同时生成的结果具有透明性和可解释性。通过输出可验证的代码,该方法将EO分析转化为透明、可复现的过程。
英文摘要:
Despite recent advances in computer vision, Earth Observation (EO) analysis remains difficult to perform for the laymen, requiring expert knowledge and technical capabilities. Furthermore, many systems return black-box predictions that are difficult to audit or reproduce. Leveraging recent advances in tool LLMs, this study proposes a conversational, code-generating agent that transforms natural-language queries into executable, auditable Python workflows. The agent operates over a unified easily extendable API for classification, segmentation, detection (oriented bounding boxes), spectral indices, and geospatial operators. With our proposed framework, it is possible to control the results at three levels: (i) tool-level performance on public EO benchmarks; (ii) at the agent-level to understand the capacity to generate valid, hallucination-free code; and (iii) at the task-level on specific use cases. In this work, we select two use-cases of interest: land-composition mapping and post-wildfire damage assessment. The proposed agent outperforms general-purpose LLM/VLM baselines (GPT-4o, LLaVA), achieving 64.2% vs. 51.7% accuracy on land-composition and 50% vs. 0% on post-wildfire analysis, while producing results that are transparent and easy to interpret. By outputting verifiable code, the approach turns EO analysis into a transparent, reproducible process.
A Multimodal Conversational Assistant for the Characterization of Agricultural Plots from Geospatial Open Data
Dept. AI \& Data Analytics CTIC Technology Center Gijón, Spain;Dept. IT CTIC Technology Center Gijón, Spain;Eng. University of Oviedo Gijón, Spain;Dept. R \& D CTIC Technology Center Gijón, Spain
RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Computing, Clemson University(克莱姆森大学计算机学院);Washington University School of Medicine in St. Louis(圣路易斯华盛顿大学医学院);The Keck School of Medicine of the University of Southern California(南加州大学凯克医学院);Mayo Clinic(梅奥诊所);Banner Alzheimer’s Institute(班纳阿尔茨海默病研究所)
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
University of Central Florida(中佛罗里达大学);University of Western Australia(西澳大学);Amazon Web Services(亚马逊网络服务);The University of Texas at Dallas(德克萨斯大学达拉斯分校)
SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation
Universitat de València(瓦伦西亚大学);Wageningen University & Research(瓦赫宁根大学与研究所);Delft University of Technology(代尔夫特理工大学);European Space Agency(欧洲航天局);Heidelberg University(海德堡大学);Ghent University(根特大学);University of Oxford(牛津大学);Google Research(谷歌研究)
An Interpretable Implicit-Based Approach for Modeling Local Spatial Effects: A Case Study of Global Gross Primary Productivity
College of Urban and Environmental Sciences, Peking University(北京大学城市与环境学院);The Hong Kong Polytechnic University(香港理工大学);School of Safety Science, Tsinghua University(清华大学安全科学学院);College of Surveying and Geo-Informatics, Tongji University(同济大学测绘与地理信息学院);School of Architecture and Urban Planning, Shenzhen University(深圳大学建筑与城市规划学院)
Earth Observation Foundation Models for Terrestrial Ecohydrology: From Representation Learning to Process Inference
The University of Sydney(悉尼大学);Helmholtz Centre for Environmental Research–UFZ(亥姆霍兹环境研究中心);Leipzig University(莱比锡大学);City University of Hong Kong(香港城市大学);University of California, Berkeley(加州大学伯克利分校);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)