WANLI：Worker和AI协作创建自然语言推理数据集
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
摘要：
大规模众包NLP数据集的一个反复出现的挑战是，人类作者在制作示例时经常依赖重复的模式，导致缺乏语言多样性。我们介绍了一种基于工人和AI协作的数据集创建新方法，该方法将语言模型的生成能力和人类的评估能力结合在一起。从一个现有的数据集，MultiNLI的自然语言推理（NLI），我们的方法使用数据集制图自动识别的例子，证明具有挑战性的推理模式，并指示GPT-3组成新的例子与类似的模式。然后机器生成的示例会自动过滤，最后由人类众包工作者进行修改和标记。由此产生的数据集WANLI由107，885个NLI示例组成，并在现有的NLI数据集上呈现出独特的经验优势。值得注意的是，与在4倍大的MultiNLI上训练相比，在WANLI上训练模型可以提高我们考虑的8个域外测试集的性能，包括在HANS上提高11%，在对抗性NLI上提高9%。此外，它仍然比其他NLI数据集增强的MultiNLI更有效。我们的研究结果证明了利用自然语言生成技术和重新想象人类在数据集创建过程中的作用的前景。
英文摘要：
A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation based on worker and AI collaboration, which brings together the generative strength of language models and the evaluative strength of humans. Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns. Machine generated examples are then automatically filtered, and finally revised and labeled by human crowdworkers. The resulting dataset, WANLI, consists of 107,885 NLI examples and presents unique empirical strengths over existing NLI datasets. Remarkably, training a model on WANLI improves performance on eight out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI, compared to training on the 4x larger MultiNLI. Moreover, it continues to be more effective than MultiNLI augmented with other NLI datasets. Our results demonstrate the promise of leveraging natural language generation techniques and re-imagining the role of humans in the dataset creation process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
