在认知不确定性下使用分位数函数上的随机过程选择拟合模型
Selecting fitted models under epistemic uncertainty using a stochastic process on quantile functions
AI总结:
针对多个模型拟合效果相同时的选择问题,提出一种基于分位数函数随机过程的非参数方法,估计复制非平稳下的经验风险不确定性,确保仅在模型可重复更优时才拒绝,优于现有标准。
摘要:
将模型拟合到数据是科学实践的重要组成部分。机器学习的进步使得拟合更多且更复杂的模型成为可能,但也加剧了一个问题:当多个模型对数据的拟合效果相同时,我们应该选择哪一个(或哪些)?答案完全取决于建模目标。在科学背景下,基本目标是可复制性:如果一个模型能很好地描述一个实验,那么当该实验在明天或在另一个实验室重复时,它应继续表现良好。因此,选择标准必须对复制过程固有的变异具有稳健性。在这项工作中,我们开发了一种非参数方法,用于在复制非平稳时估计模型经验风险的不确定性,从而确保仅当另一个模型可重复地更优时才拒绝当前模型。我们用两个例子说明该方法:一个是更经典的设置,其中模型在结构上不同;另一个是受机器学习启发的设置,其中模型仅在参数值上有所不同。我们展示了在这种可复制性或“认知不确定性”的背景下,该方法与现有模型选择标准相比表现更优,并且在大型实验数据集上具有更令人满意的行为。
英文摘要:
Fitting models to data is an important part of the practice of science. Advances in machine learning have made it possible to fit more -- and more complex -- models, but have also exacerbated a problem: when multiple models fit the data equally well, which one(s) should we pick? The answer depends entirely on the modelling goal. In the scientific context, the essential goal is _replicability_: if a model works well to describe one experiment, it should continue to do so when that experiment is replicated tomorrow, or in another laboratory. The selection criterion must therefore be robust to the variations inherent to the replication process. In this work we develop a nonparametric method for estimating uncertainty on a model's empirical risk when replications are non-stationary, thus ensuring that a model is only rejected when another is _reproducibly_ better. We illustrate the method with two examples: one a more classical setting, where the models are structurally distinct, and a machine learning-inspired setting, where they differ only in the value of their parameters. We show how, in this context of replicability or "epistemic uncertainty", it compares favourably to existing model selection criteria, and has more satisfactory behaviour with large experimental datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。