arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.09148cs.CLcs.AI
收藏

是什么让指令学习困难?综合环境下的研究与新挑战

What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment

Allen Institute for AI(艾伦人工智能研究所)

Matthew Finlayson, Kyle Richardson, Ashish Sabharwal, Peter Clark

摘要:

指令学习范式--模型仅从任务描述中学习执行新任务--在通用模型研究中变得流行。然而,大型Transformer模型作为指令学习者的能力仍然知之甚少。我们使用受控的合成环境来表征这种能力。具体来说,我们使用决定给定字符串是否匹配正则表达式(视为指令)的任务来识别使指令学习具有挑战性的任务、指令和实例的属性。例如,我们发现我们的模型,一个微调的基于T5的text 2 text Transformer,与大型正则语言斗争,这表明不太精确的指令对模型来说是一个挑战。另外,需要跟踪先前步骤的较长上下文的指令执行也更困难。我们使用我们的研究结果系统地构建了一个具有挑战性的指令学习数据集,我们称之为硬RegSet。通过对硬RegSet进行微调,我们的大型Transformer只能正确解释65.6%的测试指令(至少90%的准确率),以及分布外泛化设置中11%-24%的指令。我们提出了硬RegSet作为一个具有挑战性的指令学习任务,和一个可控的环境研究指令学习。

英文摘要:

The instruction learning paradigm -- where a model learns to perform new tasks from task descriptions alone -- has become popular in general-purpose model research. The capabilities of large transformer models as instruction learners, however, remain poorly understood. We use a controlled synthetic environment to characterize such capabilities. Specifically, we use the task of deciding whether a given string matches a regular expression (viewed as an instruction) to identify properties of tasks, instructions, and instances that make instruction learning challenging. For instance, we find that our model, a fine-tuned T5-based text2text transformer, struggles with large regular languages, suggesting that less precise instructions are challenging for models. Additionally, instruction executions that require tracking longer contexts of prior steps are also more difficult. We use our findings to systematically construct a challenging instruction learning dataset, which we call Hard RegSet. Fine-tuning on Hard RegSet, our large transformer learns to correctly interpret only 65.6% of test instructions (with at least 90% accuracy), and 11%-24% of the instructions in out-of-distribution generalization settings. We propose Hard RegSet as a challenging instruction learning task, and a controlled environment for studying instruction learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑