ClimateX：大语言模型能否准确评估人类专家对气候陈述的置信度？
ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?
AI总结：
本文构建了包含8094条IPCC气候陈述的ClimateX数据集，发现大语言模型在少样本下最多只能以47%的准确率分类专家置信度，且普遍对低中置信度陈述过度自信。
摘要：
评估大语言模型（LLMs）生成输出的准确性在气候科学与政策领域尤为重要。我们引入了专家气候陈述置信度（ClimateX）数据集，这是一个新颖的、经过人工整理和专家标注的数据集，包含从最新的政府间气候变化专门委员会（IPCC）报告中收集的8094条气候陈述，并标注了相应的置信度水平。利用该数据集，我们表明近期的大语言模型能够对气候相关陈述中的人类专家置信度进行分类，尤其是在少样本学习设置下，但准确率有限（最高为47%）。总体而言，模型对低置信度和中等置信度陈述表现出一致且显著的过度自信。我们强调了研究结果对气候传播、大语言模型评估策略以及大语言模型在信息检索系统中应用的启示。
英文摘要：
Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
