RESPONSE：基准测试语言模型在危机情境中进行常识推理的能力
RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation
AI总结：
本文构建人工标注数据集RESPONSE，从多时间阶段评估LLM的灾害常识推理，发现GPT-4等模型在即时行动上的正确率仅37%。
摘要：
当人们面临自然灾害时，会出现一类有趣的常识推理问题。为研究这一主题，我们提出了 \ extsf{RESPONSE}，这是一个由人工整理的数据集，包含1789个标注实例，涵盖6037组问题，旨在评估LLM在不同时间框架下于灾害情境中的常识推理能力。该数据集包括问题描述、缺失资源、具有时效性的解决方案及其理由，其中一个子集由环境工程师验证。通过自动指标和人工评估，我们将LLM生成的建议与人类回答进行比较。研究结果表明，即使是像GPT-4这样的最先进模型，在即时响应行动上的人工评估正确率也仅达到37%，凸显出LLM在危机中常识推理能力仍有显著提升空间。
英文摘要：
An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present \textsf{RESPONSE}, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37\% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
