arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.19630cs.LG
收藏

大规模基于Transformer的天气预报训练配方分析与探索

Analyzing and Exploring Training Recipes for Large-Scale Transformer-Based Weather Prediction

Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);University of California, Berkeley(加州大学伯克利分校);Indiana University Bloomington(印第安纳大学伯明顿分校)

Jared D. Willard, Peter Harrington, Shashank Subramanian, Ankur Mahesh, Travis A. O'Brien, William D. Collins

AI总结:

本文通过最小修改的SwinV2 Transformer在ERA5数据上训练,以简单训练流程和适度算力实现了优于IFS的预报技巧,并通过消融实验系统分析了损失函数、模型规模、深度及多步微调等训练配方对预报性能的影响。

摘要:

深度学习在数值天气预报中的迅速崛起,催生了大量模型,这些模型对大气变量的预报技巧可与传统基于物理的数值天气预报相当或更优。然而,在这些领先的深度学习模型中,训练设置和架构存在很大差异。此外,由于缺乏全面的消融研究,很难辨别哪些组件对成功最为关键。在本工作中,我们表明,即使使用相对现成的架构、简单的训练流程和适度的计算预算,也有可能获得较高的预报技巧。具体而言,我们在ERA5数据上训练了一个仅做最小修改的SwinV2 Transformer,发现其预报技巧优于IFS。我们对训练流程的关键方面进行了消融研究,探索了不同的损失函数、模型规模和深度以及多步微调,以考察它们的影响。我们还使用超出典型ACC和RMSE的指标检验了模型性能,并研究了性能如何随模型规模扩展。

英文摘要:

The rapid rise of deep learning (DL) in numerical weather prediction (NWP) has led to a proliferation of models which forecast atmospheric variables with comparable or superior skill than traditional physics-based NWP. However, among these leading DL models, there is a wide variance in both the training settings and architecture used. Further, the lack of thorough ablation studies makes it hard to discern which components are most critical to success. In this work, we show that it is possible to attain high forecast skill even with relatively off-the-shelf architectures, simple training procedures, and moderate compute budgets. Specifically, we train a minimally modified SwinV2 transformer on ERA5 data, and find that it attains superior forecast skill when compared against IFS. We present some ablations on key aspects of the training pipeline, exploring different loss functions, model sizes and depths, and multi-step fine-tuning to investigate their effect. We also examine the model performance with metrics beyond the typical ACC and RMSE, and investigate how the performance scales with model size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑