arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.03588cs.LGstat.ME
收藏

M5预测竞赛评价体系初探

A Look at the Evaluation Setup of the M5 Forecasting Competition

Monash University(莫纳什大学);University of Sydney(悉尼大学)

Hansika Hewamalage, Pablo Montero-Manso, Christoph Bergmeir, Rob J Hyndman

摘要:

预测评估在经验证据如何塑造学科发展方面起着关键作用。领域专家对与他们的决策需求相关的错误度量感兴趣。这些措施可能产生不可靠的结果。虽然已经讨论了几个度量的可靠性属性,但它几乎没有以客观的方式量化。我们提出了一个名为秩稳定性的度量,它评估了当模型和误差恒定时,实验的排名在相似数据集之间的差异。我们使用它来研究M5的评估设置。我们发现,M5的评估设置是不可靠的比其他措施。不稳定的主要驱动因素是层级聚集和规模扩大。价格加权降低了所有测试误差测量的稳定性。M5误差测量的尺度归一化导致比其他无尺度误差更低的稳定性。单独采取的层次结构水平不太稳定,聚合程度更高,它们的组合甚至比单个水平更不稳定。我们还展示了在不影响稳定性的情况下保留聚合重要性的积极权衡。聚合和稳定性可以与备受争议的幻数的影响联系起来。我们的许多研究结果可以应用到一般的层次预测基准。

英文摘要:

Forecast evaluation plays a key role in how empirical evidence shapes the development of the discipline. Domain experts are interested in error measures relevant for their decision making needs. Such measures may produce unreliable results. Although reliability properties of several metrics have already been discussed, it has hardly been quantified in an objective way. We propose a measure named Rank Stability, which evaluates how much the rankings of an experiment differ in between similar datasets, when the models and errors are constant. We use this to study the evaluation setup of the M5. We find that the evaluation setup of the M5 is less reliable than other measures. The main drivers of instability are hierarchical aggregation and scaling. Price-weighting reduces the stability of all tested error measures. Scale normalization of the M5 error measure results in less stability than other scale-free errors. Hierarchical levels taken separately are less stable with more aggregation, and their combination is even less stable than individual levels. We also show positive tradeoffs of retaining aggregation importance without affecting stability. Aggregation and stability can be linked to the influence of much debated magic numbers. Many of our findings can be applied to general hierarchical forecast benchmarking.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑