基于大型语言模型的危机报告中因果证据提取与三角验证：一项基于ReliefWeb的研究
Causal Evidence Extraction and Triangulation in Crisis Reports using Large Language Models: A ReliefWeb-based Study
浏览论文内容
中文总结 AI 辅助
该研究基于ReliefWeb数据，提出两阶段LLM流水线提取危机报告中结构化因果证据，结合上下文保留的三角验证方法，在100份专家标注报告中取得优异性能，为现金援助的食品相关结果提供了高收敛性证据。
中文摘要 AI 辅助
人道主义报告冗长、嘈杂且多主题，难以整合与决策相关的因果证据。我们开展了一项基于ReliefWeb（2000-2024年）的研究，提出了一个两阶段大型语言模型（LLM）流水线，用于提取带有方向和强度属性的结构化干预-结果记录。基于查询的提取将输出限制在指定的干预类别内，减少了检索导致的过度提取；而片段 grounding 则将每个关系链接到支持文本，以实现可审计性和分类。在一个包含100份报告的专家标注数据集中，性能最佳的闭源LLM取得了90.73%的加权F1分数，且成本效益优异；经过监督微调的Llama-3.1-8B则达到了94.15%的加权F1分数。我们进一步提出了保留上下文的三角验证方法，该方法在灾害×来源单元格内聚合强度加权的证据，应用拉普拉斯平滑并对单元格赋予同等权重，通过证据等级（Level-of-Evidence）分数量化跨上下文的收敛性。将该方法应用于现金援助，与食品相关的结果呈现出强烈的正收敛性（证据等级=0.865）和稳定的长期轨迹。
英文摘要
Humanitarian reports are long, noisy, and multi-topic, making it difficult to consolidate decision-relevant causal evidence. We present a ReliefWeb study (2000-2024) and a two-stage Large Language Model (LLM) pipeline that extracts structured intervention-outcome records with direction and strength attributes. Query-conditioned extraction restricts output to a specified intervention class, reducing retrieval-induced over-extraction, while snippet grounding links each relation to supporting text for auditability and classification. In an expert-annotated dataset of 100 reports, the best closed-source LLM achieved a weighted F1 score of 90.73% with strong cost-efficiency, while Llama-3.1-8B with supervised fine-tuning reached 94.15% weighted F1 score. We further propose context-preserving triangulation that aggregates strength-weighted evidence within disaster$\times$source cells, applies Laplace smoothing and equally weights cells to quantify cross-context convergence via a Level-of-Evidence score. Applied to cash assistance, food-related outcomes show strong positive convergence (LoE=0.865) and stable long-horizon trajectories.
发表机构
- University of Oulu（奥卢大学）
- LUT University（拉普兰塔理工大学）
机构由 AI 辅助整理，请以论文原文为准。
