WXImpactBench：一个用于评估大语言模型的破坏性天气影响理解基准
WXImpactBench: A Disruptive Weather Impact Understanding Benchmark for Evaluating Large Language Models
AI总结：
本研究构建了破坏性天气影响数据集，提出首个评估大语言模型相关理解能力的基准WXImpactBench，通过多标签分类和排序问答任务开展实验，为气候适应系统开发提供参考，相关资源已公开。
摘要：
适应气候变化需要理解破坏性天气对社会的影响，大语言模型（LLMs）或许可应用于该领域，但由于高质量语料收集难度大且缺乏可用基准，其有效性尚未得到充分探究。地方报纸中存储的气候相关事件记录了社区如何适应灾害并从中恢复，但原始语料的处理并非易事。本研究中，我们首先通过四阶段精心设计的构建流程，开发了一个破坏性天气影响数据集。随后，我们提出WXImpactBench——首个用于评估LLMs对破坏性天气影响理解能力的基准，该基准包含两项评估任务：多标签分类和基于排序的问答。通过对一组LLMs开展广泛实验，我们为开发破坏性天气影响理解与气候变化适应系统所面临的挑战提供了一手分析。构建的数据集及评估框架代码已公开，以助力社会防范灾害带来的脆弱性。
英文摘要：
Climate change adaptation requires the understanding of disruptive weather impacts on society, where large language models (LLMs) might be applicable. However, their effectiveness is under-explored due to the difficulty of high-quality corpus collection and the lack of available benchmarks. The climate-related events stored in regional newspapers record how communities adapted and recovered from disasters. However, the processing of the original corpus is non-trivial. In this study, we first develop a disruptive weather impact dataset with a four-stage well-crafted construction pipeline. Then, we propose WXImpactBench, the first benchmark for evaluating the capacity of LLMs on disruptive weather impacts. The benchmark involves two evaluation tasks, multi-label classification and ranking-based question answering. Extensive experiments on evaluating a set of LLMs provide first-hand analysis of the challenges in developing disruptive weather impact understanding and climate change adaptation systems. The constructed dataset and the code for the evaluation framework are available to help society protect against vulnerabilities from disasters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
