AANG:自动化辅助学习
AANG: Automating Auxiliary Learning
摘要:
辅助目标,即辅助学习信号,被引入以帮助在数据匮乏或高度复杂的最终任务上进行学习,在机器学习中很常见。虽然已经做了很多工作来制定有用的辅助目标,但它们的构建仍然是一种缓慢而繁琐的手工设计的艺术。关于这些目标如何以及何时提高最终任务绩效的直觉也有有限的理论支持。在这项工作中,我们提出了一种自动生成一套辅助目标的方法。我们通过在一个新的统一分类中解构现有的目标,识别它们之间的联系,并根据未发现的结构生成新的目标来实现这一目标。接下来,我们从理论上形式化了关于辅助学习如何提高最终任务泛化能力的广泛直觉。这导致我们的原则和有效的算法搜索空间生成的目标,找到那些最有用的一个指定的最终任务。以自然语言处理(NLP)作为我们的研究领域,我们证明了我们的自动辅助学习管道在5个NLP任务的预训练模型上的持续训练实验中,比竞争基线有了很大的改进。
英文摘要:
Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whilst much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand-design. Intuition for how and when these objectives improve end-task performance has also had limited theoretical backing. In this work, we present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. Next, we theoretically formalize widely-held intuitions about how auxiliary learning improves generalization on the end-task. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task. With natural language processing (NLP) as our domain of study, we demonstrate that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。