面向文档驱动气候决策的负责任检索增强生成
Responsible Retrieval Augmented Generation for Climate Decision Making from Documents
AI总结：
针对气候文档信息难获取及生成式AI的幻觉、可控性差、领域性能低等问题，提出面向气候领域的RAG评估框架，发布人工标注数据集与自动化工具，为负责任部署气候决策RAG系统及UX设计提供依据。
摘要：
气候决策受限于冗长、专业且多语言文档中关键信息的复杂性与难获取性。生成式AI技术为提升这类文档所含信息的可及性提供了可行路径，但存在诸多局限：包括（1）易产生幻觉或错误表述信息；（2）难以引导或保证生成输出的属性；（3）在特定技术领域性能下降。为应对这些挑战，我们提出了一个全新的评估框架，该框架配备了针对气候相关文档定制的领域特定维度。随后我们应用该框架评估检索增强生成（Retrieval-Augmented Generation, RAG）方法，并在一个可解答单份气候法律与政策文档相关问题的原型工具中，评估检索质量与生成质量。此外，我们发布了一个人工标注数据集以及可扩展的自动化评估工具，旨在推动这类系统在气候领域的更广泛应用与稳健评估。我们的研究结果明确了负责任部署RAG以提升决策能力的关键组成部分，同时也为高风险领域中安全部署此类系统、建立用户信任所需的用户体验（UX）考量提供了洞见。
英文摘要：
Climate decision making is constrained by the complexity and inaccessibility of key information within lengthy, technical, and multi-lingual documents. Generative AI technologies offer a promising route for improving the accessibility of information contained within these documents, but suffer from limitations. These include (1) a tendency to hallucinate or mis-represent information, (2) difficulty in steering or guaranteeing properties of generated output, and (3) reduced performance in specific technical domains. To address these challenges, we introduce a novel evaluation framework with domain-specific dimensions tailored for climate-related documents. We then apply this framework to evaluate Retrieval-Augmented Generation (RAG) approaches and assess retrieval- and generation-quality within a prototype tool that answers questions about individual climate law and policy documents. In addition, we publish a human-annotated dataset and scalable automated evaluation tools, with the aim of facilitating broader adoption and robust assessment of these systems in the climate domain. Our findings highlight the key components of responsible deployment of RAG to enhance decision-making, while also providing insights into user experience (UX) considerations for safely deploying such systems to build trust with users in high-risk domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
