Loss minimization is a dominant paradigm in machine learning, where a predictor is trained to minimize some loss function that depends on an uncertain event (e.g., "will it rain tomorrow?''). Different loss functions imply different learning algorithms and, at times, very different predictors. While widespread and appealing, a clear drawback of this approach is that the loss function may not be known at the time of learning, requiring the algorithm to use a best-guess loss function. We suggest a rigorous new paradigm for loss minimization in machine learning where the loss function can be ignored at the time of learning and only be taken into account when deciding an action.
We introduce the notion of an (${\mathcal{L}},\mathcal{C}$)-omnipredictor, which could be used to optimize any loss in a family ${\mathcal{L}}$. Once the loss function is set, the outputs of the predictor can be post-processed (a simple univariate data-independent transformation of individual predictions) to do well compared with any hypothesis from the class $\mathcal{C}$. The post processing is essentially what one would perform if the outputs of the predictor were true probabilities of the uncertain events. In a sense, omnipredictors extract all the predictive power from the class $\mathcal{C}$, irrespective of the loss function in $\mathcal{L}$.
We show that such "loss-oblivious'' learning is feasible through a connection to multicalibration, a notion introduced in the context of algorithmic fairness. In addition, we show how multicalibration can be viewed as a solution concept for agnostic boosting, shedding new light on past results. Finally, we transfer our insights back to the context of algorithmic fairness by providing omnipredictors for multi-group loss minimization.
不变风险最小化(Invariant Risk Minimization,ARJovsky et al.,2019)是最近提出的一个框架,旨在学习预测器,这些预测器在不同的训练环境中对虚假相关性是不变的。然而,尽管有理论上的理由,但在各种环境中,这种方法并没有得到广泛的测试。为了更好地理解该框架,我们使用IRMv1实证研究了几个研究问题,IRMv1是第一个提出来近似解决问题的实用算法。通过以不同的方式扩展ColoredMNIST实验,我们发现IRMv1(i)在训练环境之间的虚假相关性变化更大时表现更好,(ii)当底层关系近似不变时学习近似不变的预测器,(iii)可以扩展到文本分类的类似设置。
Probable Domain Generalization via Quantile Risk Minimization
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Edinburgh(爱丁堡大学);University of Cambridge(剑桥大学);University of Pennsylvania(宾夕法尼亚大学)
Department of Mathematical Sciences University of Copenhagen(数学科学系,哥本哈根大学);ETH Zürich(苏黎世联邦理工学院);Seminar for Statistics ETH Zürich(统计系,苏黎世联邦理工学院);ETH AI Center(苏黎世联邦理工学院人工智能中心);Biogeochemical Integration Department Max-Planck-Institute for Biogeochemistry(生物地球化学整合部门,马克斯·普朗克生物地球化学研究所)