arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.09054cs.CL
收藏

通过自然语言可满足性推动Transformers中规则推理的极限

Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiability

Allen Institute for AI(艾伦人工智能研究所)

Kyle Richardson, Ashish Sabharwal

摘要:

研究Transformer模型的推理能力,并为它们发现新的挑战性任务,一直是人们感兴趣的话题。最近的研究发现,这些模型在对用自然语言表达的形式逻辑理论进行演绎推理时表现得出奇的强大。然而,这些研究的一个缺点是,它们没有考虑到逻辑理论,当随机均匀抽样时,不一定会导致硬实例。我们提出了一种新的方法来创建具有挑战性的算法推理数据集,专注于自然语言可满足性(NLSat)的问题。其核心思想是从命题SAT难题的经验抽样和语言的复杂性理论研究中汲取见解。这种方法使我们能够区分容易和困难的实例,并系统地增加现有的推理基准,如RuleTaker的复杂性。我们发现,电流Transformers,有足够的训练数据,是令人惊讶的强大的解决所产生的NLSat问题的难度大大增加。它们还表现出一定程度的尺度不变性--能够推广到更大规模和范围的问题。然而,我们的研究结果也揭示了重要的局限性:对训练数据进行仔细的采样对于构建可推广到更大问题的模型至关重要,而Transformer模型有限的尺度不变性表明它们远不能学习鲁棒的演绎推理算法。

英文摘要:

Investigating the reasoning abilities of transformer models, and discovering new challenging tasks for them, has been a topic of much interest. Recent studies have found these models to be surprisingly strong at performing deductive reasoning over formal logical theories expressed in natural language. A shortcoming of these studies, however, is that they do not take into account that logical theories, when sampled uniformly at random, do not necessarily lead to hard instances. We propose a new methodology for creating challenging algorithmic reasoning datasets that focus on natural language satisfiability (NLSat) problems. The key idea is to draw insights from empirical sampling of hard propositional SAT problems and from complexity-theoretic studies of language. This methodology allows us to distinguish easy from hard instances, and to systematically increase the complexity of existing reasoning benchmarks such as RuleTaker. We find that current transformers, given sufficient training data, are surprisingly robust at solving the resulting NLSat problems of substantially increased difficulty. They also exhibit some degree of scale-invariance - the ability to generalize to problems of larger size and scope. Our results, however, reveal important limitations too: a careful sampling of training data is crucial for building models that generalize to larger problems, and transformer models' limited scale-invariance suggests they are far from learning robust deductive reasoning algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑