面向环境审查与许可的大语言模型基准测试
Benchmarking LLMs for Environmental Review and Permitting
AI总结：
本研究针对大语言模型在NEPA环境监管领域的应用空白，推出首个EIS文档问答基准NEPAQuAD与评估流水线MAPLE，测试发现LLM长上下文推理能力不足，RAG方法优于直接PDF上下文。
摘要：
《国家环境政策法》（NEPA）是美国的一部基础性环境立法，要求联邦机构考量其拟议行动的环境影响。实现这一要求的主要机制是编制环境评估（EAs），若存在重大影响则需编制全面的环境影响报告书（EIS）。大语言模型（LLM）在NEPA这类专业领域中的有效性尚未经过验证，无法确定能否应用于联邦决策流程。为填补这一空白，我们提出了NEPA问答数据集（NEPAQuAD）——首个源自EIS文档的综合基准，同时推出模块化且透明的评估流水线MAPLE，用于评估LLM在聚焦NEPA的监管推理任务上的表现。我们的基准利用真实EIS文档构建了多种问题类型，涵盖事实类到复杂问题解决类。我们搭建了模块化、透明的评估流水线，可在零样本或上下文驱动的问答基准中测试闭源与开源模型。我们使用该框架评估了5款先进LLM，考察它们的先验知识以及处理NEPA特定信息的能力。实验结果显示，所有模型在以黄金段落作为上下文时，性能均 consistently 达到最高。对比各模型的其他上下文驱动方法时，基于检索增强生成（RAG）的方法表现大幅优于PDF文档上下文，这表明两类模型都不擅长长上下文问答任务。我们的分析表明，聚焦NEPA的监管推理任务对LLM构成了重大挑战，尤其是在理解复杂语义以及有效处理冗长监管文档方面。
英文摘要：
The National Environment Policy Act (NEPA) stands as a foundational piece of environmental legislation in the United States, requiring federal agencies to consider the environmental impacts of their proposed actions. The primary mechanism for achieving this is through the preparation of Environmental Assessments (EAs) and, for significant impacts, comprehensive Environmental Impact Statements (EIS). Large Language Model (LLM)s' effectiveness in specialized domains like NEPA remains untested for adoption in federal decision-making processes. To address this gap, we present NEPA Question and Answering Dataset (NEPAQuAD), the first comprehensive benchmark derived from EIS documents, along with a modular and transparent evaluation pipeline, MAPLE, to assess LLM performance on NEPA-focused regulatory reasoning tasks. Our benchmark leverages actual EIS documents to create diverse question types, ranging from factual to complex problem-solving ones. We built a modular and transparent evaluation pipeline to test both closed- and open-source models in zero-shot or context-driven QA benchmarks. We evaluate five state-of-the-art LLMs using our framework to assess both their prior knowledge and their ability to process NEPA-specific information. The experimental results reveal that all the models consistently achieve their highest performance when provided with the gold passage as context. While comparing the other context-driven approaches for each model, Retrieval Augmented Generation (RAG)-based approaches substantially outperform PDF document contexts, indicating that neither model is well suited for long-context question-answering tasks. Our analysis suggests that NEPA-focused regulatory reasoning tasks pose a significant challenge for LLMs, particularly in terms of understanding the complex semantics and effectively processing the lengthy regulatory documents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
