大语言模型(LLM)正被用于数据科学代码生成任务,但它们在处理复杂的顺序任务时往往遇到困难,从而导致逻辑错误。由于难以纳入复杂的数据结构和空间约束、难以有效利用多样化的函数调用,并且倾向于幻觉出较少使用的地理空间库,因此将其应用于地理空间数据处理尤其具有挑战性。为解决这些问题,我们提出 GeoAgent,这是一种新的交互式框架,旨在帮助 LLM 更有效地处理地理空间数据。GeoAgent 率先在 Monte Carlo Tree Search(MCTS,蒙特卡洛树搜索)算法中集成代码解释器、静态分析和 Retrieval-Augmented Generation(RAG,检索增强生成)技术,为地理空间数据处理提供了一种新颖方法。此外,我们还贡献了一个专门用于评估基于 LLM 的地理空间任务方法的新基准。该基准利用多种 Python 库,包含数据获取、数据分析和可视化等单轮与多轮任务。通过在多样化地理空间语境中提供全面评估,该基准为开发面向地理空间数据分析任务的基于 LLM 的方法设立了新标准。我们的研究结果表明,仅依赖 LLM 的知识不足以准确完成地理空间任务编程,因为这类任务需要连贯的多步骤流程和多次函数调用。与基线 LLM 相比,所提出的 GeoAgent 展现出更优性能,在函数调用和任务完成方面取得显著提升。此外,这些结果为未来在自动化地理空间数据分析任务编程中开发 LLM 智能体提供了宝贵见解。
英文摘要:
Large language models (LLMs) are being used in data science code generation tasks, but they often struggle with complex sequential tasks, leading to logical errors. Their application to geospatial data processing is particularly challenging due to difficulties in incorporating complex data structures and spatial constraints, effectively utilizing diverse function calls, and the tendency to hallucinate less-used geospatial libraries. To tackle these problems, we introduce GeoAgent, a new interactive framework designed to help LLMs handle geospatial data processing more effectively. GeoAgent pioneers the integration of a code interpreter, static analysis, and Retrieval-Augmented Generation (RAG) techniques within a Monte Carlo Tree Search (MCTS) algorithm, offering a novel approach to geospatial data processing. In addition, we contribute a new benchmark specifically designed to evaluate the LLM-based approach in geospatial tasks. This benchmark leverages a variety of Python libraries and includes both single-turn and multi-turn tasks such as data acquisition, data analysis, and visualization. By offering a comprehensive evaluation among diverse geospatial contexts, this benchmark sets a new standard for developing LLM-based approaches in geospatial data analysis tasks. Our findings suggest that relying solely on knowledge of LLM is insufficient for accurate geospatial task programming, which requires coherent multi-step processes and multiple function calls. Compared to the baseline LLMs, the proposed GeoAgent has demonstrated superior performance, yielding notable improvements in function calls and task completion. In addition, these results offer valuable insights for the future development of LLM agents in automatic geospatial data analysis task programming.
Risk-Aware LLM Agents for Geospatial Data Retrieval: Design and Preliminary Adversarial Evaluation
Dept. of Systems Design Engineering, University of Waterloo(滑铁卢大学系统设计工程系);SkyWatch;Dept. of Geography and Environmental Management, University of Waterloo(滑铁卢大学地理与环境管理系);Dept. of Geomatics Engineering, University of Calgary(卡尔加里大学测绘工程系)
Evaluation of Code LLMs on Geospatial Code Generation
Wrocław University of Science and Technology(弗罗茨瓦夫理工大学);INESC-ID;LUMLIS (Lisbon ELLIS Unit)(LUMLIS(里斯本ELLIS单元));Instituto Superior Técnico(高等技术学院);University of Lisbon(里斯本大学)
GeoGPT: Understanding and Processing Geospatial Tasks through An Autonomous GPT
School of Geography and Information Engineering, China University of Geosciences(中国地质大学地理与信息工程学院);National Engineering Research Center for Geographic Information System, China University of Geosciences(中国地质大学地理信息系统国家工程研究中心)
AutoGEEval: A Multimodal and Automated Framework for Geospatial Code Generation on GEE with Large Language Models
Wuhan University(武汉大学);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University(武汉大学测绘遥感信息工程国家重点实验室);Collaborative Innovation Center of Geospatial Technology, Wuhan University(武汉大学地理空间技术协同创新中心);School of Resource and Environmental Sciences, Wuhan University(武汉大学资源与环境科学学院);School of Remote Sensing and Information Engineering, Wuhan University(武汉大学遥感信息工程学院)
Securing Multi-Agent GIS Systems: Risk Evaluation and Prompt Hardening Optimization
Department of Systems Design Engineering, University of Waterloo(系统设计工程系,滑铁卢大学);Department of Mechatronics Engineering, University of Waterloo(机电工程系,滑铁卢大学);Department of Geomatics Engineering, University of Calgary(测绘工程系,卡尔加里大学);Department of Geography and Environmental Management, University of Waterloo(地理与环境管理系,滑铁卢大学)
Using Generative AI and Multi-Agents to Provide Automatic Feedback
School of Teacher Education, Nanjing Normal University(南京师范大学教师教育学院);AI4STEM Education Center, University of Georgia(佐治亚大学AI4STEM教育中心);National GENIUS Center, University of Georgia(佐治亚大学国家GENIUS中心);Department of Mathematics, Science, and Social Studies Education, University of Georgia(佐治亚大学数学、科学与社会研究教育系);Beijing Institute of Education(北京教育学院)
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Zhejiang University of Technology(浙江工业大学)
STEAMROLLER: A Multi-Agent System for Inclusive Automatic Speech Recognition for People who Stutter
Shanghai Key Laboratory of Trustworthy Computing, East China Normal University(华东师范大学可信计算上海市重点实验室);AI Research Team, Quantstamp(Quantstamp AI研究团队);The University of New South Wales(新南威尔士大学);Nanyang Technological University(南洋理工大学);Huazhong University of Science and Technology(华中科技大学)
Can AI automatically analyze public opinion? A LLM agents-based agentic pipeline for timely public opinion analysis
Brunel Design School, Brunel University of London(布鲁内尔设计学院,伦敦布鲁内尔大学);College of Engineering, Design and Physical Sciences, Brunel University of London(工程、设计与物理科学学院,伦敦布鲁内尔大学);Faculty of Science and Engineering, University of Hull(科学与工程学院,赫尔大学)