arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.02892cs.CLcs.LG
收藏

子任务分解支持按顺序学习以排序任务

Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks

The Hebrew University of Jerusalem(耶路撒冷希伯来大学)

Noam Wies, Yoav Levine, Amnon Shashua

摘要:

自然语言处理领域的能力经历了巨大的飞跃,最近推出了巨大的语言模型。尽管取得了这样的成功,涉及几个复杂步骤的自然语言问题实际上仍然是不可学习的,即使是最大的LM。这符合在各种领域中证明的复合问题的端到端学习的实验失败。一种有效的缓解方法是引入中间监督来解决复合问题的子任务。最近,一些作品已经表现出高收益,通过采取一种简单的方法,将中间监督在复合自然语言问题:序列到序列LM馈与增强输入,其中分解的任务的标签简单地连接到原始输入。在本文中,我们证明了一个积极的学习结果,激励这些最近的努力。我们表明,当连接中间监督的输入和训练的序列到序列模型在此修改后的输入,不可学习的复合问题可以成为学习。我们表明,这是真实的任何家庭的任务,一方面,是不可学习的,另一方面,可以分解成一个多项式数量的简单的子任务,其中每一个只依赖于O(1)以前的子任务的结果。除了激励当代将中间监督纳入序列到序列语言模型的经验努力之外,我们的积极理论结果是关于中间监督对神经网络学习的好处的结果景观中的第一个:到目前为止,关于这个主题的所有理论结果都是负面的,即,显示的情况下,学习是不可能的,没有中间监督,而我们的结果是积极的,表明学习是促进在中间监督的存在。

英文摘要:

The field of Natural Language Processing has experienced a dramatic leap in capabilities with the recent introduction of huge Language Models. Despite this success, natural language problems that involve several compounded steps are still practically unlearnable, even by the largest LMs. This complies with experimental failures for end-to-end learning of composite problems that were demonstrated in a variety of domains. An effective mitigation is to introduce intermediate supervision for solving sub-tasks of the compounded problem. Recently, several works have demonstrated high gains by taking a straightforward approach for incorporating intermediate supervision in compounded natural language problems: the sequence-to-sequence LM is fed with an augmented input, in which the decomposed tasks' labels are simply concatenated to the original input. In this paper, we prove a positive learning result that motivates these recent efforts. We show that when concatenating intermediate supervision to the input and training a sequence-to-sequence model on this modified input, unlearnable composite problems can become learnable. We show that this is true for any family of tasks which on the one hand, are unlearnable, and on the other hand, can be decomposed into a polynomial number of simple sub-tasks, each of which depends only on O(1) previous sub-task results. Beyond motivating contemporary empirical efforts for incorporating intermediate supervision in sequence-to-sequence language models, our positive theoretical result is the first of its kind in the landscape of results on the benefits of intermediate supervision for neural-network learning: Until now, all theoretical results on the subject are negative, i.e., show cases where learning is impossible without intermediate supervision, while our result is positive, showing that learning is facilitated in the presence of intermediate supervision.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑