ClimaQA:一个面向气候问答模型的自动化评估框架
ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models
AI总结:
本文提出ClimaQA评估框架,通过ClimaGen生成专家标注与大规模合成气候问答数据集,并系统比较不同LLMs在气候科学问答上的表现。
摘要:
大型语言模型(LLMs)在气候科学中的应用近年来受到了广泛关注。然而,一个关键问题仍然存在:缺乏一个能够评估模型输出质量和科学有效性的综合评估框架。为了解决这一问题,我们开发了ClimaGen(Climate QA Generator),这是一个自适应学习框架,在气候科学家的参与下,从研究生教材中生成问答对。由此,我们提出了ClimaQA-Gold,一个专家标注的基准数据集,以及ClimaQA-Silver,一个面向气候科学的大规模综合合成问答数据集。最后,我们制定了评估策略,并在我们的基准上比较了不同的LLMs。我们的结果为增强气候LLMs知识所采用的各种方法提供了新的见解。源代码已在 https://github.com/Rose-STL-Lab/genie-climaqa 公开。
英文摘要:
The use of Large Language Models (LLMs) in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an adaptive learning framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different LLMs on our benchmarks. Our results offer novel insights into various approaches used to enhance knowledge of climate LLMs. The source code is publicly available at https://github.com/Rose-STL-Lab/genie-climaqa
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。