时间序列预测中的“挑拣”:如何选择数据集使你的模型大放异彩
Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine
AI总结:
本文研究了时间序列预测中“挑拣”数据集对性能评估的扭曲影响,发现增加测试数据集数量可显著降低误判算法最优性的风险,呼吁采用更全面的评估框架。
摘要:
时间序列预测的重要性推动了针对该问题的持续研究与开发新方法。通常,这些方法通过实证研究引入,并经常声称所提出的方法具有更高的准确率。然而,由于实验设置的局限性,这些结果的可靠性与泛化性引发了越来越多的担忧。本文探讨了一个关键局限:所使用数据集的数量与代表性。我们研究了数据集选择偏差的影响,特别是“挑拣”数据集的做法对预测方法性能评估的影响。通过对一组多样化的基准数据集进行实证分析,我们的研究结果表明,挑拣数据集会显著扭曲对方法性能的感知,常常夸大其有效性。此外,我们的结果显示,通过仅选择4个数据集(大多数研究报告的数量),46%的方法可被认为是同类最佳,77%的方法可跻身前三名。另外,近期基于深度学习的方法对数据集选择表现出高敏感性,而经典方法则展现出更强的鲁棒性。最后,我们的结果表明,在基准子集上实证验证预测算法时,将测试数据集数量从3个增加到6个,可将错误识别某算法为最佳算法的风险降低约40%。本研究强调了对能更准确反映现实场景的全面评估框架的迫切需求。采用此类框架将确保开发出稳健且可靠的预测方法。
英文摘要:
The importance of time series forecasting drives continuous research and the development of new approaches to tackle this problem. Typically, these methods are introduced through empirical studies that frequently claim superior accuracy for the proposed approaches. Nevertheless, concerns are rising about the reliability and generalizability of these results due to limitations in experimental setups. This paper addresses a critical limitation: the number and representativeness of the datasets used. We investigate the impact of dataset selection bias, particularly the practice of cherry-picking datasets, on the performance evaluation of forecasting methods. Through empirical analysis with a diverse set of benchmark datasets, our findings reveal that cherry-picking datasets can significantly distort the perceived performance of methods, often exaggerating their effectiveness. Furthermore, our results demonstrate that by selectively choosing just four datasets - what most studies report - 46% of methods could be deemed best in class, and 77% could rank within the top three. Additionally, recent deep learning-based approaches show high sensitivity to dataset selection, whereas classical methods exhibit greater robustness. Finally, our results indicate that, when empirically validating forecasting algorithms on a subset of the benchmarks, increasing the number of datasets tested from 3 to 6 reduces the risk of incorrectly identifying an algorithm as the best one by approximately 40%. Our study highlights the critical need for comprehensive evaluation frameworks that more accurately reflect real-world scenarios. Adopting such frameworks will ensure the development of robust and reliable forecasting methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。