arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.03068cs.LGcs.AIstat.ML
收藏

通过风险分解评估自监督学习

Evaluating Self-Supervised Learning via Risk Decomposition

Stanford University(斯坦福大学)

Yann Dubois, Tatsunori Hashimoto, Percy Liang

AI总结:

提出一种SSL风险分解方法,将误差分为四个组件以分析设计选择对169个模型的影响,为改进SSL提供洞见。

摘要:

自监督学习(SSL)流程在架构、数据增强或预训练数据等许多设计选择上存在差异。然而,SSL通常使用单一指标进行评估:在ImageNet上进行线性探测。这并不能提供太多关于模型为何或何时更好的洞见,也无法说明如何改进它。为了解决这个问题,我们提出了一种SSL风险分解方法,它通过考虑表征学习步骤中产生的误差,推广了经典的监督近似-估计分解。我们的分解包含四个误差组件:近似、表征可用性、探测泛化和编码器泛化。我们为每个组件提供了高效的估计器,并用它们分析了在ImageNet上评估的169个SSL视觉模型中30种设计选择的影响。我们的分析为设计和使用SSL模型提供了有价值的洞见。例如,它突出了误差的主要来源,并展示了如何通过权衡误差组件在特定设置(全样本 vs 少样本)下改进SSL。所有结果和预训练模型均可在 https://github.com/YannDubs/SSL-Risk-Decomposition 获取。

英文摘要:

Self-supervised learning (SSL) pipelines differ in many design choices such as the architecture, augmentations, or pretraining data. Yet SSL is typically evaluated using a single metric: linear probing on ImageNet. This does not provide much insight into why or when a model is better, now how to improve it. To address this, we propose an SSL risk decomposition, which generalizes the classical supervised approximation-estimation decomposition by considering errors arising from the representation learning step. Our decomposition consists of four error components: approximation, representation usability, probe generalization, and encoder generalization. We provide efficient estimators for each component and use them to analyze the effect of 30 design choices on 169 SSL vision models evaluated on ImageNet. Our analysis gives valuable insights for designing and using SSL models. For example, it highlights the main sources of error and shows how to improve SSL in specific settings (full- vs few-shot) by trading off error components. All results and pretrained models are at https://github.com/YannDubs/SSL-Risk-Decomposition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑