Training data for text classification is often limited in practice, especially for applications with many output classes or involving many related classification problems. This means classifiers must generalize from limited evidence, but the manner and extent of generalization is task dependent. Current practice primarily relies on pre-trained word embeddings to map words unseen in training to similar seen ones. Unfortunately, this squishes many components of meaning into highly restricted capacity. Our alternative begins with sparse pre-trained representations derived from unlabeled parsed corpora; based on the available training data, we select features that offers the relevant generalizations. This produces task-specific semantic vectors; here, we show that a feed-forward network over these vectors is especially effective in low-data scenarios, compared to existing state-of-the-art methods. By further pairing this network with a convolutional neural network, we keep this edge in low data scenarios and remain competitive when using full training sets.
Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets
Bar-Ilan University(巴伊兰大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学)
现象特定的“对抗性”数据集最近被设计为针对特定推理类型执行有针对性的压力测试。最近的工作(Liu et al.,2019 a)提出,这样的数据集可以用于训练NLI和其他类型的模型,通常允许学习焦点现象并改进挑战数据集,这表明原始训练数据中存在“盲点”。然而,尽管模型可以在这样的训练过程中得到改进,但它仍然可能容易受到针对相同现象但来自不同分布的其他挑战数据集的影响,例如具有不同的语法复杂性水平。在这项工作中,我们扩展了这种方法,通过控制对抗数据集中的其他方面,来得出关于模型学习和概括目标现象的能力的结论,而不是“学习”数据集。我们展示了我们的方法上的两个推理现象-与格交替和数值推理,阐述,并在某些情况下矛盾,刘等人的结果。我们的方法能够构建更好的挑战数据集,以创建更强大的模型,并可能产生更好的模型理解和随后的总体改进。
Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models
Harvard University(哈佛大学);Massachusetts Institute of Technology(麻省理工学院);UC Irvine(加州大学欧文分校);IBM Research(IBM研究院);Amazon AI(亚马逊人工智能)
人类可以从最少的经验中学习一个词的结构属性,并在不同的语法上下文中统一部署他们学习的句法表示。我们评估了现代神经语言模型在英语中再现这种行为的能力,并评估了结构监督对学习结果的影响。首先,我们通过开发控制实验来评估少样本学习能力,该实验探测模型的句法名义数和言语论元结构概括,用于在训练过程中只看到两次的标记。其次,我们评估了学习表征的不变性:模型从基本上下文(例如,简单的陈述性主动语态句子)到变换的上下文(例如,疑问句)。我们测试了在同一数据集上训练的四个模型:一个n-gram基线,一个LSTM,以及两个用显式结构监督训练的LSTM变体(Dyer et al.,2016; Charniak等人,2016)。我们发现,在大多数情况下,神经模型能够在最小的暴露后诱导出正确的语法概括,通常在训练过程中只使用两个示例,并且这两个结构监督模型比LSTM模型更准确地概括。所有的神经模型都能够利用在基本上下文中学习到的信息来驱动转换后的上下文中的期望,这表明它们已经学习了语法的一些不变性。
Mix and Match: Context Pairing for Scalable Topic-Controlled Educational Summarisation
Visual Intelligence Laboratory, Department of Statistics, Faculty of Science, Khon Kaen University, Khon Kaen 40002, Thailand(科纳坎大学科学学院统计学系视觉智能实验室);Centre for Artificial Intelligence, Department of Computer Science, University College London, London WC1E 6BT, The United Kingdom(伦敦大学学院人工智能中心计算机科学系)