arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.09377cs.CLcs.AIcs.LG
收藏

伪对数似然法在自然语言评分中的应用

An Application of Pseudo-Log-Likelihoods to Natural Language Scoring

Dalhousie University(达尔豪斯大学);Brock University(布鲁克大学)

Darren Abramson, Ali Emami

摘要:

使用半监督机器学习在大型自然语言语料库上构建的语言模型已经非常迅速地涵盖了自然语言生成和理解的领域。在本文中,我们采用了zero-shot的方法独立开发的一些研究人员现在获得认可,作为一个重要的替代微调评价常识的任务。与较新的语言模型(T5)相比,具有相对较少参数和训练步骤的语言模型可以在最近的大型数据集(TimeDial)上表现出色,同时在类似的语言任务类别中表现出稳健性。令人惊讶的是,与微调到较大模型相比,通过使用具有较小模型的无超参数zero-shot方法来实现该结果。我们认为,更小的模型的鲁棒性应该被理解的组合性,在某种意义上说,我们从最近的文献中一类类似的模型。我们确定了我们的方法和模型的实际成本:自然语言评估的高GPU时间。对于ALBERT和其他BERT变体,产生显著稳定性的zero-shot测量技术是伪对数似然性对掩蔽语言模型的应用,用于在诸如Winograd Schema Challenge、Winogrande等的强制选择语言任务中替代替代方案的概率的相对测量。本文的一个贡献是汇集了一些类似的,但独立的研究链。我们在二元选择任务中产生了一些绝对最先进的常识推理结果,表现优于文献中任何已发表的结果,包括微调的努力。我们展示了该模型在对抗性环境下的表现的显著一致性,我们认为这最好由该模型的组合性表征来解释。

英文摘要:

Language models built using semi-supervised machine learning on large corpora of natural language have very quickly enveloped the fields of natural language generation and understanding. In this paper we apply a zero-shot approach independently developed by a number of researchers now gaining recognition as a significant alternative to fine-tuning for evaluation on common sense tasks. A language model with relatively few parameters and training steps compared to a more recent language model (T5) can outperform it on a recent large data set (TimeDial), while displaying robustness in its performance across a similar class of language tasks. Surprisingly, this result is achieved by using a hyperparameter-free zero-shot method with the smaller model, compared to fine-tuning to the larger model. We argue that robustness of the smaller model ought to be understood in terms of compositionality, in a sense that we draw from recent literature on a class of similar models. We identify a practical cost for our method and model: high GPU-time for natural language evaluation. The zero-shot measurement technique that produces remarkable stability, both for ALBERT and other BERT variants, is an application of pseudo-log-likelihoods to masked language models for the relative measurement of probability for substitution alternatives in forced choice language tasks such as the Winograd Schema Challenge, Winogrande, and others. One contribution of this paper is to bring together a number of similar, but independent strands of research. We produce some absolute state-of-the-art results for common sense reasoning in binary choice tasks, performing better than any published result in the literature, including fine-tuned efforts. We show a remarkable consistency of the model's performance under adversarial settings, which we argue is best explained by the model's compositionality of representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑