避免元摊销误差的Meta循环退火算法
Meta Cyclical Annealing Schedule: A Simple Approach to Avoiding Meta-Amortization Error
摘要：
用少量数据学习新概念的能力是智能的一个重要方面，这对深度学习方法来说是一个挑战。针对Few-Shot学习的元学习为这个问题提供了一个潜在的解决方案：通过学习从许多以前的任务中跨数据学习，Few-Shot学习算法可以发现任务之间的结构，从而能够快速学习新任务。然而，Few-Shot学习中的一个关键挑战是任务模糊性：即使可以从大量先前任务中元学习到强大的先验，新任务的小数据集也可能非常模糊，无法为该任务获取单个模型。贝叶斯元学习模型通过引入一个复杂的先验分布，并利用贝叶斯决策理论对后验分布进行正则化，很好地解决了这个问题。然而，目前已知的贝叶斯元学习过程，如VERSA遭受所谓的{ \it信息偏好问题}，即，后验分布退化到一个点，并且远离精确的一个。为了解决这个问题，我们设计了一个新的元正则化目标使用{ \it循环退火时间表}和{ \it最大平均差异}（MMD）标准。循环退火时间表在避免这种退化解方面非常有效。这个过程包括一个困难的KL发散估计，但我们解决了这个问题，采用MMD而不是KL发散。实验结果表明，我们的方法大大优于标准的元学习算法。
英文摘要：
The ability to learn new concepts with small amounts of data is a crucial aspect of intelligence that has proven challenging for deep learning methods. Meta-learning for few-shot learning offers a potential solution to this problem: by learning to learn across data from many previous tasks, few-shot learning algorithms can discover the structure among tasks to enable fast learning of new tasks. However, a critical challenge in few-shot learning is task ambiguity: even when a powerful prior can be meta-learned from a large number of prior tasks, a small dataset for a new task can simply be very ambiguous to acquire a single model for that task. The Bayesian meta-learning models can naturally resolve this problem by putting a sophisticated prior distribution and let the posterior well regularized through Bayesian decision theory. However, currently known Bayesian meta-learning procedures such as VERSA suffer from the so-called {\it information preference problem}, that is, the posterior distribution is degenerated to one point and is far from the exact one. To address this challenge, we design a novel meta-regularization objective using {\it cyclical annealing schedule} and {\it maximum mean discrepancy} (MMD) criterion. The cyclical annealing schedule is quite effective at avoiding such degenerate solutions. This procedure includes a difficult KL-divergence estimation, but we resolve the issue by employing MMD instead of KL-divergence. The experimental results show that our approach substantially outperforms standard meta-learning algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
