敏感度作为序列分类任务的复杂性度量
Sensitivity as a Complexity Measure for Sequence Classification Tasks
摘要:
我们引入了一个理论框架,用于理解和预测序列分类任务的复杂性,使用布尔函数灵敏度理论的新扩展。给定输入序列的分布,函数的灵敏度量化输入序列的不相交子集的数量,每个子集可以单独改变以改变输出。我们认为,标准的序列分类方法偏向于学习低灵敏度的功能,因此,需要高灵敏度的任务更加困难。为此,我们分析表明,简单的词汇分类器只能表达有限的灵敏度的功能,我们的经验表明,低灵敏度的功能更容易学习LSTM。然后,我们估计15个NLP任务的敏感性,发现在GLUE中收集的具有挑战性的任务的敏感性高于简单的文本分类任务,并且该敏感性预测了简单词汇分类器和香草BiLSTM的性能,而无需预先训练的上下文嵌入。在一项任务中,敏感性预测了哪些输入对于这些简单的模型来说是困难的。我们的研究结果表明,大规模预训练的上下文表示的成功部分源于它们提供的表示,可以通过低敏感度解码器提取信息。
英文摘要:
We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。