arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.02598cs.LGcs.CV
收藏

预测不是理解:在机器学习中识别和解决欠规范

Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning

Idiap Research Institute(Idiap研究所);EPFL(洛桑联邦理工学院);Australian Institute for Machine Learning(澳大利亚机器学习研究所)

Damien Teney, Maxime Peyrard, Ehsan Abbasnejad

摘要:

机器学习(ML)模型通常针对其在给定数据集上的准确性进行优化。然而,这种预测标准很少能捕捉到模型的所有理想属性,特别是它与领域专家对任务的理解的匹配程度。欠规范是指存在多个模型,这些模型在其域内准确性方面无法区分,即使它们在其他期望属性(如分布外(OOD)性能)方面存在差异。识别这些情况对于评估ML模型的可靠性至关重要。我们形式化的欠规格的概念,并提出了一种方法来识别和部分解决它。我们训练多个模型的独立约束,迫使他们实现不同的功能。他们发现了标准经验风险最小化(ERM)所忽略的预测特征,然后我们将其提取到具有卓越OOD性能的全局模型中。重要的是,我们约束模型与数据流形对齐,以确保它们发现有意义的特征。我们在计算机视觉(拼贴画,WILDS-Camelyon 17,GQA)的多个数据集上演示了该方法,并讨论了欠规范的一般含义。最值得注意的是,如果没有额外的假设,域内性能不能用于OOD模型选择。

英文摘要:

Machine learning (ML) models are typically optimized for their accuracy on a given dataset. However, this predictive criterion rarely captures all desirable properties of a model, in particular how well it matches a domain expert's understanding of a task. Underspecification refers to the existence of multiple models that are indistinguishable in their in-domain accuracy, even though they differ in other desirable properties such as out-of-distribution (OOD) performance. Identifying these situations is critical for assessing the reliability of ML models. We formalize the concept of underspecification and propose a method to identify and partially address it. We train multiple models with an independence constraint that forces them to implement different functions. They discover predictive features that are otherwise ignored by standard empirical risk minimization (ERM), which we then distill into a global model with superior OOD performance. Importantly, we constrain the models to align with the data manifold to ensure that they discover meaningful features. We demonstrate the method on multiple datasets in computer vision (collages, WILDS-Camelyon17, GQA) and discuss general implications of underspecification. Most notably, in-domain performance cannot serve for OOD model selection without additional assumptions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑