AgentCaster：推理引导的龙卷风预报
AgentCaster: Reasoning-Guided Tornado Forecasting
AI总结：
该研究提出无数据污染的多模态LLM龙卷风预报框架AgentCaster及专属评估指标，发现人类专家预报表现远超易幻觉、过估风险的现有SOTA模型，助力关键领域LLM智能体推理能力提升研究。
摘要：
目前人们越来越需要在复杂、高影响的现实任务上评估大语言模型（LLM），以判断其作为推理智能体的真实就绪程度。为填补这一空白，我们提出AgentCaster——一个无数据污染的框架，采用多模态大语言模型端到端执行具有挑战性的长周期龙卷风预报任务。在AgentCaster中，模型解读来自高分辨率对流可分辨预报档案的异质时空数据。我们在为期40天的多样历史数据上评估模型性能，数据涵盖数次重大龙卷风爆发事件，包含500多份龙卷风报告。每天，模型可从3625张预报图和40125份预报探空数据中交互式查询，预报时效为12-36小时。模型输出的概率龙卷风风险多边形预测结果，会与在投影坐标空间中对不相交风险带进行几何比较得到的真值进行验证。为量化准确率，我们提出了领域专属的TornadoBench和TornadoHallucination指标，其中TornadoBench对大语言模型和领域人类预报专家都极具挑战性。值得注意的是，人类专家的表现显著优于当前最先进的模型；这些模型表现出强烈的幻觉倾向和风险强度过度预测问题，难以实现精准的地理定位，且在复杂动态演化系统中的时空推理能力较差。AgentCaster旨在推动改进大语言模型智能体的研究，以应对关键领域中具有挑战性的推理任务。
英文摘要：
There is a growing need to evaluate Large Language Models (LLMs) on complex, high-impact, real-world tasks to assess their true readiness as reasoning agents. To address this gap, we introduce AgentCaster, a contamination-free framework employing multimodal LLMs end-to-end for the challenging, long-horizon task of tornado forecasting. Within AgentCaster, models interpret heterogeneous spatiotemporal data from a high-resolution convection-allowing forecast archive. We assess model performance over a 40-day period featuring diverse historical data, spanning several major tornado outbreaks and including over 500 tornado reports. Each day, models query interactively from a pool of 3,625 forecast maps and 40,125 forecast soundings for a forecast horizon of 12-36 hours. Probabilistic tornado-risk polygon predictions are verified against ground truths derived from geometric comparisons across disjoint risk bands in projected coordinate space. To quantify accuracy, we propose domain-specific TornadoBench and TornadoHallucination metrics, with TornadoBench highly challenging for both LLMs and domain expert human forecasters. Notably, human experts significantly outperform state-of-the-art models, which demonstrate a strong tendency to hallucinate and overpredict risk intensity, struggle with precise geographic placement, and exhibit poor spatiotemporal reasoning in complex, dynamically evolving systems. AgentCaster aims to advance research on improving LLM agents for challenging reasoning tasks in critical domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
