一阶 ANIL 在过参数化下可证明地学习表示
First-order ANIL provably learns representations despite overparametrization
AI总结:
本工作证明,在任务数无限极限下,使用线性双层网络的一阶 ANIL 即便在过参数化下也能成功学习线性共享表示,并取得渐近低秩解,从而在单次梯度步后对新任务实现良好适应。
摘要:
由于在少样本分类和强化学习中的经验成功,元学习最近受到了广泛关注。元学习方法利用先前任务的数据,以样本高效的方式学习新任务。具体而言,模型不可知方法寻找初始化点,使得梯度下降能快速适应任何新任务。尽管经验表明此类方法通过在预训练期间学习共享表示而表现良好,但关于这种行为的理论证据有限。更重要的是,尚未证明这些方法在存在架构错误设定时仍能学习共享结构。在此方向上,本工作表明,在任务数量无限多的极限下,使用线性双层网络架构的一阶 ANIL 成功学习了线性共享表示。该结果甚至在过参数化情况下依然成立;宽度大于共享表示维度会导致渐近低秩解。学习到的解在单次梯度步后即可在任何新任务上产生良好的适应性能。总体而言,这说明了诸如一阶 ANIL 等模型不可知方法如何良好地学习共享表示。
英文摘要:
Due to its empirical success in few-shot classification and reinforcement learning, meta-learning has recently received significant interest. Meta-learning methods leverage data from previous tasks to learn a new task in a sample-efficient manner. In particular, model-agnostic methods look for initialization points from which gradient descent quickly adapts to any new task. Although it has been empirically suggested that such methods perform well by learning shared representations during pretraining, there is limited theoretical evidence of such behavior. More importantly, it has not been shown that these methods still learn a shared structure, despite architectural misspecifications. In this direction, this work shows, in the limit of an infinite number of tasks, that first-order ANIL with a linear two-layer network architecture successfully learns linear shared representations. This result even holds with overparametrization; having a width larger than the dimension of the shared representations results in an asymptotically low-rank solution. The learned solution then yields a good adaptation performance on any new task after a single gradient step. Overall, this illustrates how well model-agnostic methods such as first-order ANIL can learn shared representations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。