元学习的分布依赖分析
A Distribution-Dependent Analysis of Meta-Learning
摘要:
元学习理论中的一个关键问题是理解任务分布如何影响迁移风险,即元学习者对未知任务分布的新任务的预期错误。在本文中,专注于固定设计的线性回归高斯噪声和高斯任务(或参数)分布,我们给分布依赖的下限上的任何算法的转移风险,同时我们也表明,一种新的,加权版本的所谓的有偏正则化回归方法是能够匹配这些下限到一个固定的常数因子。值得注意的是,加权是从高斯任务分布的协方差导出的。总而言之,我们的研究结果提供了一个精确的描述在这种高斯设置的元学习的难度。虽然这个问题的设置可能看起来很简单,我们表明,它是足够丰富的统一的“参数共享”和“表示学习”流的元学习,特别是,表示学习作为特殊情况下,任务分布的协方差阵是未知的。对于这种情况下,我们建议采用EM方法,这是在我们的情况下,享受有效的更新。最后,本文通过对EM的实证研究来完成。特别是,我们的实验结果表明,期望最大化算法可以达到下界的任务数量的增长,而该算法也是成功的竞争时,其替代品用于表示学习的上下文中。
英文摘要:
A key problem in the theory of meta-learning is to understand how the task distributions influence transfer risk, the expected error of a meta-learner on a new task drawn from the unknown task distribution. In this paper, focusing on fixed design linear regression with Gaussian noise and a Gaussian task (or parameter) distribution, we give distribution-dependent lower bounds on the transfer risk of any algorithm, while we also show that a novel, weighted version of the so-called biased regularized regression method is able to match these lower bounds up to a fixed constant factor. Notably, the weighting is derived from the covariance of the Gaussian task distribution. Altogether, our results provide a precise characterization of the difficulty of meta-learning in this Gaussian setting. While this problem setting may appear simple, we show that it is rich enough to unify the "parameter sharing" and "representation learning" streams of meta-learning; in particular, representation learning is obtained as the special case when the covariance matrix of the task distribution is unknown. For this case we propose to adopt the EM method, which is shown to enjoy efficient updates in our case. The paper is completed by an empirical study of EM. In particular, our experimental results show that the EM algorithm can attain the lower bound as the number of tasks grows, while the algorithm is also successful in competing with its alternatives when used in a representation learning context.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。