Recent years have seen a surge of interest in meta-learning techniques for tackling the few-shot learning (FSL) problem. However, the meta-learner is prone to overfitting since there are only a few available samples, which can be identified as sampling noise on a clean dataset. Moreover, when handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise on a corrupted dataset. To address these two challenges, we present Eigen-Reptile (ER) that updates the meta-parameters with the main direction of historical task-specific parameters to alleviate sampling and label noise. Specifically, the main direction is computed in a fast way, where the scale of the calculated matrix is related to the number of gradient steps instead of the number of parameters. Furthermore, to obtain a more accurate main direction for Eigen-Reptile in the presence of many noisy labels, we further propose Introspective Self-paced Learning (ISPL). We have theoretically and experimentally demonstrated the soundness and effectiveness of the proposed Eigen-Reptile and ISPL. Particularly, our experiments on different tasks show that the proposed method is able to outperform or achieve highly competitive performance compared with other gradient-based methods with or without noisy labels. The code and data for the proposed method are provided for research purposes https://github.com/Anfeather/Eigen-Reptile.
École Polytechnique de Montréal(蒙特利尔综合理工学院);McGill University(麦吉尔大学)
生成对抗网络(GAN)拥有令人印象深刻的生成逼真图像的能力。然而,与深度学习领域的许多领域一样,它们需要大量的数据才能产生结果,从而限制了它们在产生新颖性方面的有用性。同样,元学习的最新进展为许多少样本学习应用打开了大门。在目前的工作中,我们提出了Few-Shot Image Generation using Reptile(FIGR),一个使用Reptile进行元训练的GAN。我们的模型成功地在MNIST和Omniglot上生成了新的图像,其中只有4张来自看不见的类的图像。我们进一步贡献了FIGR-8,这是一个用于Few-Shot图像生成的新数据集,其中包含1,548,944个图标,分为18,409个类别。在FIGR-8上进行训练,初步结果表明,我们的模型可以从以前看不见的图像类别中的8个样本和这8个图像中的10个训练步骤中概括出更高级的概念(如“鸟”和“刀”)。这项工作展示了训练GAN用于Few-Shot图像生成的潜力,并旨在为该领域的未来工作设定新的基准。
许多现实世界的问题,包括多说话者文本到语音合成,可以大大受益于元学习大型模型的能力,只有几个特定于任务的组件。只更新这些特定于任务的模块,然后允许模型适应低数据任务,并根据需要进行尽可能多的步骤,而不会冒过拟合的风险。不幸的是,现有的元学习方法要么不能扩展到长期适应,要么依赖于手工制作的特定于任务的架构。在这里,我们提出了一个元学习的方法,避免了这种往往次优的手选择的需要。特别是,我们开发了基于贝叶斯收缩的通用技术,以自动发现和学习特定任务和通用可重用模块。从经验上讲,我们证明我们的方法发现了一小部分有意义的特定于任务的模块,并且在任务数据很少和适应范围较长的少量文本到语音(Few-Shot text to speech)等领域中优于现有的元学习方法。我们还表明,现有的元学习方法,包括MAML,iMAML和爬行动物出现我们的方法的特殊情况。
Adaptive Clipping for Privacy-Preserving Few-Shot Learning: Enhancing Generalization with Limited Data
Deakin University(迪肯大学);Data61, CSIRO(联邦科学与工业研究组织Data61研究院);The University of Alabama in Huntsville(阿拉巴马大学亨茨维尔分校);University of New South Wales (UNSW)(新南威尔士大学)
A Unified Information-Theoretic Framework for Meta-Learning Generalization
School of Computer Science of Technology(科技学院计算机科学系);School of Biomedical Informatics(生物医学信息学系);Department of Oncology(肿瘤学系);Department of Computer Science and Technology(计算机科学与技术系)