结构多样抽样的样本有效训练与综合评价
Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation
摘要：
越来越多的研究表明，NLP模型无法在组成上进行泛化，并试图通过专门的架构、训练方案和数据增强等方法来缓解这一问题。在这项工作中，我们研究了一种不同的方法：在具有不同结构的实例上进行训练。我们提出了一个模型不可知的算法，从一个标记的实例池结构化输出的二次抽样这样的实例集。评估两个组成的模板分裂和传统的IID分裂的5个不同复杂度的语义解析数据集，我们表明，结构上不同的训练使用我们的算法导致可比或更好的泛化比以前的算法在9个10个小块分裂类型对。在一般情况下，我们发现结构多样性，以不断提高样本效率相比，随机训练集。此外，我们表明，结构多样的抽样产生全面的测试集，比IID测试集更具挑战性。最后，我们从不同的训练集提供了两个解释，提高泛化：1）提高覆盖率的输出子结构，和2）减少这些子结构之间的虚假相关性。
英文摘要：
A growing body of research has demonstrated the inability of NLP models to generalize compositionally and has tried to alleviate it through specialized architectures, training schemes, and data augmentation, among other approaches. In this work, we study a different approach: training on instances with diverse structures. We propose a model-agnostic algorithm for subsampling such sets of instances from a labeled instance pool with structured outputs. Evaluating on both compositional template splits and traditional IID splits of 5 semantic parsing datasets of varying complexity, we show that structurally diverse training using our algorithm leads to comparable or better generalization than prior algorithms in 9 out of 10 dataset-split type pairs. In general, we find structural diversity to consistently improve sample efficiency compared to random train sets. Moreover, we show that structurally diverse sampling yields comprehensive test sets that are a lot more challenging than IID test sets. Finally, we provide two explanations for improved generalization from diverse train sets: 1) improved coverage of output substructures, and 2) a reduction in spurious correlations between these substructures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
