arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.00451cs.LGcs.CV
收藏

关于神经坍缩在小样本学习元学习模型中的作用

On the Role of Neural Collapse in Meta Learning Models for Few-shot Learning

Carnegie Mellon University(卡内基梅隆大学);University of Michigan(密歇根大学)

Saaketh Medepalli, Naren Doraiswamy

AI总结:

本研究首次探索小样本学习元学习框架中的神经坍缩现象,在Omniglot数据集小样本设置下实验,发现学习特征存在坍缩趋势且随模型规模增大更明显,但未达完全坍缩,为理解元学习表征提供了新视角。

摘要:

面向小样本学习的元学习框架旨在学习能够凭借少量训练样本快速掌握新技能或适应新环境的模型,这使得所构建的模型仅需少量标注样本就能泛化到新类别。然而这类网络被视为黑盒模型,理解不同学习场景下学习到的表征至关重要。神经坍缩($\boldsymbol{\rm \bf \text{NC}}$)是近期发现的一种现象,展现了网络在趋近零损失过程中的独特性质:输入特征坍缩到各自的类别均值,类别均值构成单纯形等角紧框架(Simplex equiangular tight frame, ETF)——此时类别均值间距最大且线性可分,分类器则充当简单的最近邻分类器。尽管这些现象已在简单分类网络中被观测到,本研究首次探索并解析了神经坍缩在小样本学习元学习框架中的性质。我们在Omniglot数据集的小样本设置下开展研究,考察神经坍缩现象,发现学习到的特征确实存在神经坍缩趋势,且随模型规模增大该趋势更明显,但从$\boldsymbol{\rm \bf \text{NC}}$性质的衡量标准来看,并不一定会呈现完全坍缩。

英文摘要:

Meta-learning frameworks for few-shot learning aims to learn models that can learn new skills or adapt to new environments rapidly with a few training examples. This has led to the generalizability of the developed model towards new classes with just a few labelled samples. However these networks are seen as black-box models and understanding the representations learnt under different learning scenarios is crucial. Neural collapse ($\mathcal{NC}$) is a recently discovered phenomenon which showcases unique properties at the network proceeds towards zero loss. The input features collapse to their respective class means, the class means form a Simplex equiangular tight frame (ETF) where the class means are maximally distant and linearly separable, and the classifier acts as a simple nearest neighbor classifier. While these phenomena have been observed in simple classification networks, this study is the first to explore and understand the properties of neural collapse in meta learning frameworks for few-shot learning. We perform studies on the Omniglot dataset in the few-shot setting and study the neural collapse phenomenon. We observe that the learnt features indeed have the trend of neural collapse, especially as model size grows, but to do not necessarily showcase the complete collapse as measured by the $\mathcal{NC}$ properties.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑