arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.09931cs.LGcs.AIcs.CVstat.ML
收藏

解构分布:一个逐点学习框架

Deconstructing Distributions: A Pointwise Framework of Learning

Harvard(哈佛大学);UC Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学);UC San Diego(加州大学圣地亚哥分校)

Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, Preetum Nakkiran

摘要:

在机器学习中,我们传统上评估单个模型的性能,对测试输入集合进行平均。在这项工作中,我们提出了一种新的方法:我们衡量的性能时,一个$\textit{single input point}$模型的集合进行评估。具体来说,我们研究了一个点的$\textit{profile}$:模型在测试分布上的平均性能和它们在这个单独点上的逐点性能之间的关系。我们发现,配置文件可以产生新的见解模型和数据的结构-在和分布。例如,我们凭经验表明,真实数据分布由具有定性不同轮廓的点组成。一方面,在逐点和平均表现之间存在具有强相关性的“兼容”点。另一方面,存在弱相关甚至$\textit{negative}$相关的点:在这些输入上,提高整体模型精度实际上是提高了$\textit{hurts}$性能。我们证明,这些实验观察与先前工作中提出的几个简化的学习模型的预测是不一致的。作为一个应用程序,我们使用配置文件来构建一个数据集,我们称之为CIFAR-10-NEG:CINIC-10的一个子集,对于标准模型,CIFAR-10-NEG的准确性是$\textit{negatively correlated}$,CIFAR-10测试的准确性。这是第一次展示了一个完全颠倒“在线准确性”的OOD数据集(Miller,Taori,Raghunathan,Sagawa,Koh,Shankar,Liang,Carmon和Schmidt 2021)

英文摘要:

In machine learning, we traditionally evaluate the performance of a single model, averaged over a collection of test inputs. In this work, we propose a new approach: we measure the performance of a collection of models when evaluated on a $\textit{single input point}$. Specifically, we study a point's $\textit{profile}$: the relationship between models' average performance on the test distribution and their pointwise performance on this individual point. We find that profiles can yield new insights into the structure of both models and data -- in and out-of-distribution. For example, we empirically show that real data distributions consist of points with qualitatively different profiles. On one hand, there are "compatible" points with strong correlation between the pointwise and average performance. On the other hand, there are points with weak and even $\textit{negative}$ correlation: cases where improving overall model accuracy actually $\textit{hurts}$ performance on these inputs. We prove that these experimental observations are inconsistent with the predictions of several simplified models of learning proposed in prior work. As an application, we use profiles to construct a dataset we call CIFAR-10-NEG: a subset of CINIC-10 such that for standard models, accuracy on CIFAR-10-NEG is $\textit{negatively correlated}$ with accuracy on CIFAR-10 test. This illustrates, for the first time, an OOD dataset that completely inverts "accuracy-on-the-line" (Miller, Taori, Raghunathan, Sagawa, Koh, Shankar, Liang, Carmon, and Schmidt 2021)

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑