大型语言模型能否生成地理空间代码?
Can Large Language Models Generate Geospatial Code?
AI总结:
本文提出GeoCode-Eval框架和GeoCode-Bench基准数据集,系统评估LLMs在地理空间代码生成上的能力,并通过微调Code LLaMA-7B得到GEECode-GPT,证明领域数据集构建能显著提升代码生成效果。
摘要:
随着对时空数据处理和地理空间建模需求的不断增长,自动化地理空间代码生成已成为提升生产力的关键。大型语言模型(LLMs)在代码生成方面展现出潜力,但面临领域特定知识缺口和“编码幻觉”等挑战。本文提出GeoCode-Eval(GCE),一个用于评估LLMs生成地理空间代码能力的框架,该框架从三个维度进行评估:“认知与记忆”、“理解与解释”以及“创新与创造”,并分布在八个能力层级上。我们构建了一个基准数据集GeoCode-Bench,包含5,000道多项选择题、1,500道填空题、1,500道判断题以及1,000项主观任务,涵盖代码摘要、生成、补全和纠错。利用GeoCode-Bench,我们评估了三个商业闭源LLMs、四个开源通用LLMs以及14个专用代码生成模型。我们还进行了少样本与零样本学习、思维链推理以及多轮多数投票实验,以衡量它们对地理空间代码生成的影响。此外,我们使用Google Earth Engine相关的JavaScript对Code LLaMA-7B模型进行微调,创建了GEECode-GPT,并在主观任务上对其进行了评估。结果表明,构建预训练和指令数据集能显著改善代码生成,为在特定领域优化LLMs提供了见解。
英文摘要:
With the growing demand for spatiotemporal data processing and geospatial modeling, automating geospatial code generation has become essential for productivity. Large language models (LLMs) show promise in code generation but face challenges like domain-specific knowledge gaps and "coding hallucinations." This paper introduces GeoCode-Eval (GCE), a framework for assessing LLMs' ability to generate geospatial code across three dimensions: "Cognition and Memory," "Comprehension and Interpretation," and "Innovation and Creation," distributed across eight capability levels. We developed a benchmark dataset, GeoCode-Bench, consisting of 5,000 multiple-choice, 1,500 fill-in-the-blank, 1,500 true/false questions, and 1,000 subjective tasks covering code summarization, generation, completion, and correction. Using GeoCode-Bench, we evaluated three commercial closed-source LLMs, four open-source general-purpose LLMs, and 14 specialized code generation models. We also conducted experiments on few-shot and zero-shot learning, Chain of Thought reasoning, and multi-round majority voting to measure their impact on geospatial code generation. Additionally, we fine-tuned the Code LLaMA-7B model using Google Earth Engine-related JavaScript, creating GEECode-GPT, and evaluated it on subjective tasks. Results show that constructing pre-training and instruction datasets significantly improves code generation, offering insights for optimizing LLMs in specific domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。