WeatherArchive-Bench: 基于历史天气档案的检索增强推理基准测试
WeatherArchive-Bench: Benchmarking Retrieval-Augmented Reasoning for Historical Weather Archives
AI总结：
本文提出WeatherArchive-Bench，用于评估检索增强生成系统在处理历史天气档案中的能力，揭示密集检索器在历史术语上的不足及LLM对社会脆弱性与韧性概念的误判问题。
摘要：
历史天气事件档案是持久的原始记录集合，提供了关于社会如何经历和应对极端天气事件的丰富未被利用的叙述。这些定性记录提供了关于社会脆弱性和韧性的重要见解，这些见解在气象记录中很少见，对气候科学家理解社会反应具有价值。然而，其规模庞大、数字化质量嘈杂和语言古老，使将它们转化为结构化知识对气候研究具有挑战性。为此，我们引入WeatherArchive-Bench，这是首个用于评估检索增强生成（RAG）系统在历史天气档案上的基准。WeatherArchive-Bench包含两个任务：WeatherArchive-Retrieval，用于衡量系统从超过一百万份档案新闻片段中定位相关历史段落的能力，以及WeatherArchive-Assessment，用于评估大型语言模型（LLMs）能否从极端天气叙述中分类社会脆弱性和韧性指标。在稀疏、密集和重新排序检索器以及多样化的LLM集合上进行的广泛实验表明，密集检索器在历史术语上经常失败，而LLM经常错误解释脆弱性和韧性概念。这些发现突显了在复杂社会指标上推理的关键限制，并为从档案上下文中设计更稳健的气候聚焦RAG系统提供了见解。构建的数据集和评估框架可在https://anonymous.4open.science/r/WeatherArchive-Bench/上公开获取。
英文摘要：
Historical archives on weather events are collections of enduring primary source records that offer rich, untapped narratives of how societies have experienced and responded to extreme weather events. These qualitative accounts provide insights into societal vulnerability and resilience that are largely absent from meteorological records, making them valuable for climate scientists to understand societal responses. However, their vast scale, noisy digitized quality, and archaic language make it difficult to transform them into structured knowledge for climate research. To address this challenge, we introduce WeatherArchive-Bench, the first benchmark for evaluating retrieval-augmented generation (RAG) systems on historical weather archives. WeatherArchive-Bench comprises two tasks: WeatherArchive-Retrieval, which measures a system's ability to locate historically relevant passages from over one million archival news segments, and WeatherArchive-Assessment, which evaluates whether Large Language Models (LLMs) can classify societal vulnerability and resilience indicators from extreme weather narratives. Extensive experiments across sparse, dense, and re-ranking retrievers, as well as a diverse set of LLMs, reveal that dense retrievers often fail on historical terminology, while LLMs frequently misinterpret vulnerability and resilience concepts. These findings highlight key limitations in reasoning about complex societal indicators and provide insights for designing more robust climate-focused RAG systems from archival contexts. The constructed dataset and evaluation framework are publicly available at https://anonymous.4open.science/r/WeatherArchive-Bench/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
