arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15027cs.LG
收藏

将神经网络用于天气数据集的数据清洗

Using Neural Networks for Data Cleaning in Weather Datasets

University of Bristol(布里斯托尔大学)

Jack R. P. Hanslope, Laurence Aitchison

AI总结:

针对气候研究中不同天气数据集不对齐的问题,本研究用带噪声的IBTrACS标签训练神经网络映射风场与风暴位置,发现该网络具备去噪效果,表现优于原标签,为天气数据清洗提供了新方法。

摘要:

在气候科学领域,研究人员通常需要在不同数据集之间开展比对工作。但观测数据与再分析数据之间,甚至不同再分析数据集之间,都不可避免地存在不匹配问题,导致跨数据集比对面临困难。这种数据不对齐会给所有试图基于一个数据集对另一个数据集进行推断的研究带来问题。本研究以热带气旋定位为示例任务,采用ERA5(第五代欧洲中期天气预报中心再分析数据集)提供大气条件数据,IBTrACS(国际最佳路径档案集)提供风暴路径数据。研究发现,尽管两类数据的样本通常对齐度较好,但仍有相当比例(约25%)的样本对齐效果不佳。研究人员训练了一个神经网络,用于将风场映射到风暴位置;在该场景下,数据集的不对齐表现为“标签噪声”(即标注的风暴位置与底层风场不对应)。研究发现,这个仅使用IBTrACS中通常带有噪声的标签训练的神经网络具有去噪效果,以人类偏好为衡量标准,其表现甚至优于IBTrACS的标签本身。值得注意的是,这一结论甚至在训练样本上也成立——此前研究人员原本预期网络会在训练样本上过拟合IBTrACS的预测结果。

英文摘要:

In climate science, we often want to compare across different datasets. Difficulties can arise in doing this due to inevitable mismatches that arise between observational and reanalysis data, or even between different reanalyses. This misalignment can raise problems for any work that seeks to make inferences about one dataset from another. We considered tropical cyclone location as an example task with one dataset providing atmospheric conditions (ERA5) and another providing storm tracks (IBTrACS). We found that while the examples often aligned well, there were a considerable proportion (around 25%) which were not well aligned. We trained a neural network to map from the wind field to the storm location; in this setting misalignment in the datasets appears as "label noise" (i.e. the labelled storm location does not correspond to the underlying wind field). We found that this neural network trained only on the often noisy labels from IBTrACS had a denoising effect, and performed better than the IBTrACS labels themselves, as measured by human preferences. Remarkably, this even held true for training points, on which we might have expected the network to overfit to the IBTrACS predictions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑