arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2104.11559cs.CLcs.AI
收藏

优化为德语NER训练的小型BERT

Optimizing small BERTs trained for German NER

University of Rostock(罗斯托克大学);Institute of Mathematics(数学研究所);Planet AI GmbH(Planet AI 公司)

Jochen Zöllner, Konrad Sperfeld, Christoph Wick, Roger Labahn

摘要:

目前,用于训练语言模型的最广泛的神经网络架构是所谓的BERT,其导致各种自然语言处理(NLP)任务的改进。一般来说,BERT模型中的参数越多,在这些NLP任务中获得的结果就越好。不幸的是,内存消耗和训练持续时间随着这些模型的大小而急剧增加。在本文中,我们研究了较小的BERT模型的各种训练技术:我们结合了其他BERT变体(如ALBERT、RoBERTa和相对位置编码)的不同方法。此外,我们提出了两个新的微调修改,导致更好的性能:类开始结束标记和修改形式的线性链条件随机场。此外,我们还引入了全字注意力,与经典的多头注意力相比,它减少了BERT的内存使用,并导致性能的小幅提高。我们评估这些技术的五个公共德国命名实体识别(NER)的任务,其中两个介绍了本文。

英文摘要:

Currently, the most widespread neural network architecture for training language models is the so called BERT which led to improvements in various Natural Language Processing (NLP) tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP tasks. Unfortunately, the memory consumption and the training duration drastically increases with the size of these models. In this article, we investigate various training techniques of smaller BERT models: We combine different methods from other BERT variants like ALBERT, RoBERTa, and relative positional encoding. In addition, we propose two new fine-tuning modifications leading to better performance: Class-Start-End tagging and a modified form of Linear Chain Conditional Random Fields. Furthermore, we introduce Whole-Word Attention which reduces BERTs memory usage and leads to a small increase in performance compared to classical Multi-Head-Attention. We evaluate these techniques on five public German Named Entity Recognition (NER) tasks of which two are introduced by this article.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑