arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.09838cs.CVcs.AI
收藏

ClimateIQA: 一种新的数据集和基准,以推进气象异常分析中的视觉-语言模型

ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis

Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(人工智能研究所,香港科学与技术大学(广州));Thrust of Data Science and Analytics, The Hong Kong University of Science and Technology (Guangzhou)(数据科学与分析研究所,香港科学与技术大学(广州));Harvard University(哈佛大学);Peking University(北京大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Jian Chen, Peilin Zhou, Yining Hua, Dading Chong, Meng Cao, Yaowei Li, Wei Chen, Bing Zhu, Junwei Liang, Zixuan Yuan

AI总结:

ClimateIQA通过引入SPOT算法和新型数据集,提升视觉-语言模型在气象异常分析中的准确性和表现。

摘要:

气象热力图在解码极端天气现象中起着至关重要的作用,但其固有的复杂性,如不规则轮廓、无结构模式和复杂的颜色变化,为最先进的视觉-语言模型(VLMs)带来了独特的分析难题。当前最先进的模型,如GPT-4o、Qwen-VL和LLaVA 1.6,在精确的颜色识别和空间定位任务上表现不佳,导致解释不准确或不完整。为了解决这些挑战,我们引入了稀疏位置和轮廓跟踪(SPOT),一种专门设计用于处理视觉数据中不规则形状着色区域的新型算法。SPOT通过提取其空间坐标来识别和定位这些区域,使不规则形状能够获得结构化的表示。基于SPOT,我们构建了ClimateIQA,一个新型的气象视觉问答(VQA)数据集,包含26,280张高分辨率热力图和762,120条用于风速、总降水量、风寒指数和热量指数分析的指令样本。ClimateIQA通过整合空间线索、地理元数据和再分析数据来增强VLM训练,提高模型在解释和描述极端天气特征方面的准确性。此外,我们开发了Climate-Zoo,一套基于SPOT赋能的ClimateIQA微调的VLMs,显著优于现有模型在气象热力图任务中的表现。

英文摘要:

Meteorological heatmaps play a vital role in deciphering extreme weather phenomena, yet their inherent complexities marked by irregular contours, unstructured patterns, and complex color variations present unique analytical hurdles for state-of-the-art Vision-Language Models (VLMs). Current state-of-the-art models like GPT-4o, Qwen-VL, and LLaVA 1.6 struggle with tasks such as precise color identification and spatial localization, resulting in inaccurate or incomplete interpretations. To address these challenges, we introduce Sparse Position and Outline Tracking (SPOT), a novel algorithm specifically designed to process irregularly shaped colored regions in visual data. SPOT identifies and localizes these regions by extracting their spatial coordinates, enabling structured representations of irregular shapes. Building on SPOT, we construct ClimateIQA, a novel meteorological visual question answering (VQA) dataset, comprising 26,280 high-resolution heatmaps and 762,120 instruction samples for wind gust, total precipitation, wind chill index and heat index analysis. ClimateIQA enhances VLM training by incorporating spatial cues, geographic metadata, and reanalysis data, improving model accuracy in interpreting and describing extreme weather features. Furthermore, we develop Climate-Zoo, a suite of fine-tuned VLMs based on SPOT-empowered ClimateIQA, which significantly outperforms existing models in meteorological heatmap tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑