用于提升语言理解中神经元效用的跨模型对比损失
Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding
AI总结:
针对NLU模型神经元冗余、输入噪声干扰的问题,提出基于完整与消融模型对比原则的跨模型对比损失,从内部提升神经元效用,经多模型多任务实验验证有效,在小参数、长输入模型上优势显著。
摘要:
当前自然语言理解(NLU)模型的规模在模型体量和输入上下文两方面持续扩大,引入了更多隐藏神经元和输入神经元。虽然这总体上能提升平均性能,但额外的神经元并非对所有样本都能带来稳定提升。原因在于部分隐藏神经元存在冗余,且输入神经元中混杂的噪声容易干扰模型。以往研究主要通过网络剪枝、上下文选择等额外的后处理或预处理步骤,从外部降低低效用神经元的占比来规避该问题。除此之外,我们能否通过从内部提升每个神经元的效用,让模型减少冗余参数并抑制输入噪声?如果模型能高效利用神经元,那么无论消融(禁用)哪些神经元,被消融后的子模型性能都不应优于原始完整模型。基于这一模型间的对比原则,我们针对广泛任务提出了一种跨模型对比损失。对比损失本质上是基于完整模型与消融模型的任务特定损失构建的排序损失,其期望是完整模型的任务特定损失最小。我们基于5种广泛使用的预训练语言模型,在3类不同NLU任务的14个数据集上开展了大量实验,验证了对比损失的普适有效性,且发现它在参数较少或输入较长的模型上表现尤为突出。
英文摘要:
Current natural language understanding (NLU) models have been continuously scaling up, both in terms of model size and input context, introducing more hidden and input neurons. While this generally improves performance on average, the extra neurons do not yield a consistent improvement for all instances. This is because some hidden neurons are redundant, and the noise mixed in input neurons tends to distract the model. Previous work mainly focuses on extrinsically reducing low-utility neurons by additional post- or pre-processing, such as network pruning and context selection, to avoid this problem. Beyond that, can we make the model reduce redundant parameters and suppress input noise by intrinsically enhancing the utility of each neuron? If a model can efficiently utilize neurons, no matter which neurons are ablated (disabled), the ablated submodel should perform no better than the original full model. Based on such a comparison principle between models, we propose a cross-model comparative loss for a broad range of tasks. Comparative loss is essentially a ranking loss on top of the task-specific losses of the full and ablated models, with the expectation that the task-specific loss of the full model is minimal. We demonstrate the universal effectiveness of comparative loss through extensive experiments on 14 datasets from 3 distinct NLU tasks based on 5 widely used pretrained language models and find it particularly superior for models with few parameters or long input.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。