arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.04810cs.CL
收藏

上下文在极低资源语言建模中的重要性

The Importance of Context in Very Low Resource Language Modeling

Center for Language and Cognition(语言与认知中心);University of Groningen(格罗宁根大学)

Lukas Edman, Antonio Toral, Gertjan van Noord

摘要:

本文研究了非常低资源的语言模型预训练,当少于10万个句子。我们发现,在资源非常少的情况下,统计n-gram语言模型的性能优于最先进的神经模型。我们的实验表明,这主要是由于前者的重点放在当地的情况下。因此,我们引入了三种方法来提高神经模型在低资源环境中的性能,发现限制模型的自我注意力是最有效的方法,对于我们测试的语言:英语,印地语和土耳其语,NLI和POS标记等下游任务的改进高达5%。

英文摘要:

This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model's performance in the low-resource setting, finding that limiting the model's self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑