SynopticBench：在生成未来天气预报讨论上评估视觉-语言模型
SynopticBench: Evaluating Vision-Language Models on Generating Weather Forecast Discussions of the Future
AI总结：
本文提出SynopticBench数据集和SPACE评估框架，用于评估视觉-语言模型在生成天气现象描述中的性能，揭示现有评估指标的局限性。
摘要：
最近视觉-语言模型（VLMs）的进步在图像描述、报告生成等复杂多模态任务中取得了显著提升。然而，从气象数据生成文本极具挑战性，因为大气是一个快速变化的混沌系统。鉴于大气现象的复杂性，有必要验证现有VLMs在天气预报数据上的有效性。本文提出了SynopticBench，包含1,367,041个由美国国家气象局生成的区域预报讨论文本及其对应的500mb地势高度、2米温度和850mb风速图像。我们还提出了Synoptic Phenomena Alignment and Coverage Evaluation（SPACE），一种新的评估框架，可用于有效估计天气现象描述的质量。对最先进的VLMs生成预报讨论的大量实验显示了现有评估指标在此领域的敏感性，并促进了对天气和气候文本生成的进一步探索。
英文摘要：
Recent advances in visual-language models (VLMs) have led to significant improvements in a plethora of complex multimodal tasks like image captioning, report generation, and visual perception. However, generating text from meteorological data is highly challenging because the atmosphere is a chaotic system that is rapidly changing at various spatial and temporal scales. Given the complexity of atmospheric phenomena, it is critical to verifiably quantify the effectiveness of existing VLMs on weather forecasting data. In this work, we present SynopticBench, a high-quality dataset consisting of 1,367,041 text samples of Area Forecast Discussions created by the National Weather Service over the continental United States paired to images of 500mb geopotential height, 2 meter temperature, and 850mb wind velocity in weather forecasts. We also present Synoptic Phenomena Alignment and Coverage Evaluation (SPACE), a novel evaluation framework that can be used to effectively estimate the quality of text descriptions of synoptic weather phenomena. Extensive experiments on generating forecast discussions using state-of-the-art VLMs show the sensitivity of existing evaluation metrics in this domain and enable further exploration into synoptic weather and climate text generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
