arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.01275cs.LGcs.AI
收藏

自监督学习中的特征多样性

Feature diversity in self-supervised learning

Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);École Polytechnique de Montréal(蒙特利尔综合理工学院);Université de Montréal(蒙特利尔大学)

Pranshu Malviya, Arjun Vaithilingam Sudhakar

摘要:

许多关于缩放定律的研究考虑了一些基本因素,如模型大小、模型形状、数据集大小和计算能力。这些因素很容易调整,代表了任何机器学习设置的基本元素。但研究人员也采用了更复杂的因素来估计测试误差和泛化性能,具有较高的可预测性。这些因素通常特定于域或应用。例如,Chen等人(2021)主要使用特征多样性来促进同向真实转移。在以前的工作中定义了许多缩放因子,研究这些因子如何影响CNN模型自监督学习的整体泛化性能将是有趣的。个体因素如何促进泛化,包括不同的深度、宽度或提前停止的训练时期的数量?例如,更高的特征多样性是否会导致在复杂设置中保持更高的精度,而不是同步到真实的转换?这些因素是如何相互依赖的?我们发现,最后一层是整个培训中最多样化的。然而,虽然模型的测试误差随着历元的增加而减小,但其多样性下降。我们还发现多样性与模型宽度直接相关。

英文摘要:

Many studies on scaling laws consider basic factors such as model size, model shape, dataset size, and compute power. These factors are easily tunable and represent the fundamental elements of any machine learning setup. But researchers have also employed more complex factors to estimate the test error and generalization performance with high predictability. These factors are generally specific to the domain or application. For example, feature diversity was primarily used for promoting syn-to-real transfer by Chen et al. (2021). With numerous scaling factors defined in previous works, it would be interesting to investigate how these factors may affect overall generalization performance in the context of self-supervised learning with CNN models. How do individual factors promote generalization, which includes varying depth, width, or the number of training epochs with early stopping? For example, does higher feature diversity result in higher accuracy held in complex settings other than a syn-to-real transfer? How do these factors depend on each other? We found that the last layer is the most diversified throughout the training. However, while the model's test error decreases with increasing epochs, its diversity drops. We also discovered that diversity is directly related to model width.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑