测量和缩小语言模型中的组合性差距
Measuring and Narrowing the Compositionality Gap in Language Models
AI总结：
研究测量了语言模型在组合推理中的组合性差距，发现模型规模增大并不能缩小该差距，并提出了self-ask方法结合搜索引擎显著提升多跳问答准确性。
摘要：
我们调查了语言模型执行组合推理任务的能力，在这些任务中，整体解决方案取决于正确组合子问题的答案。我们测量模型能正确回答所有子问题却无法生成整体解决方案的频率，我们将这一比率称为组合性差距。我们通过提出多跳问题来评估这一比率，这些问题的答案需要组合在预训练期间不太可能被一起观察到的多个事实。在GPT-3系列模型中，随着模型规模的增加，我们展示了单跳问答性能的提升速度快于多跳性能的提升，因此组合性差距并没有缩小。这一令人惊讶的结果表明，虽然更强大的模型记忆并回忆了更多的事实知识，但它们在执行这种组合推理的能力上并没有表现出相应的提升。然后，我们演示了启发式提示（如思维链）如何通过显式推理来缩小组合性差距。我们提出了一种新方法，self-ask，进一步改进了思维链。在我们的方法中，模型在回答初始问题之前，明确地向自己提出（并回答）后续问题。最后，我们展示了self-ask的结构化提示使我们能够轻松插入搜索引擎来回答后续问题，这进一步提高了准确性。
英文摘要：
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease. This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning. We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly. We present a new method, self-ask, that further improves on chain of thought. In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question. We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
