arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.01874cs.LG
收藏

分布外泛化评估综述

A Survey on Evaluation of Out-of-Distribution Generalization

Tsinghua University(清华大学)

Han Yu, Jiashuo Liu, Xingxuan Zhang, Jiayun Wu, Peng Cui

AI总结:

本文首次系统综述 OOD 泛化评估,按测试数据可用性将研究分为性能测试、性能预测和内在属性刻画三类,并讨论预训练模型评估与未来方向。

摘要:

机器学习模型虽然日益先进,却严重依赖 IID(独立同分布)假设;由于不可避免的分布偏移,这一假设在实践中往往无法满足。这使得模型在风险敏感型应用中部署时显得脆弱且不可信赖。因此,这一重大问题催生了多个研究分支,致力于开发能够实现 Out-of-Distribution(OOD,分布外)泛化的算法。尽管已有这些努力,人们对 OOD 泛化评估的关注却少得多,而评估同样是一个复杂且基础性的问题。其目标不仅是判断模型的 OOD 泛化能力强弱,还要评估模型在何处泛化良好、在何处泛化较差。这需要刻画模型能够有效应对的分布偏移类型,并针对给定模型识别安全与有风险的输入区域。本文是首次对 OOD 评估进行全面综述的尝试。我们根据测试数据的可用性,将现有研究归纳为三种范式:OOD 性能测试、OOD 性能预测以及 OOD 内在属性刻画。此外,我们还简要讨论了预训练模型背景下的 OOD 评估。最后,我们提出了 OOD 评估未来研究中若干有前景的方向。

英文摘要:

Machine learning models, while progressively advanced, rely heavily on the IID assumption, which is often unfulfilled in practice due to inevitable distribution shifts. This renders them susceptible and untrustworthy for deployment in risk-sensitive applications. Such a significant problem has consequently spawned various branches of works dedicated to developing algorithms capable of Out-of-Distribution (OOD) generalization. Despite these efforts, much less attention has been paid to the evaluation of OOD generalization, which is also a complex and fundamental problem. Its goal is not only to assess whether a model's OOD generalization capability is strong or not, but also to evaluate where a model generalizes well or poorly. This entails characterizing the types of distribution shifts that a model can effectively address, and identifying the safe and risky input regions given a model. This paper serves as the first effort to conduct a comprehensive review of OOD evaluation. We categorize existing research into three paradigms: OOD performance testing, OOD performance prediction, and OOD intrinsic property characterization, according to the availability of test data. Additionally, we briefly discuss OOD evaluation in the context of pretrained models. In closing, we propose several promising directions for future research in OOD evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑