KinyaBERT：一种形态感知的卢旺达语语言模型
KinyaBERT: a Morphology-aware Kinyarwanda Language Model
AI总结：
针对形态丰富的低资源卢旺达语，本文提出利用形态分析器并显式建模形态组合性的双层 BERT 架构 KinyaBERT，在 NER 与翻译版 GLUE 上显著优于基线。
摘要：
诸如 BERT 之类的预训练语言模型已成功应对许多自然语言处理任务。然而，这些模型中常用的无监督子词分词方法（例如字节对编码——BPE）在处理形态丰富的语言时并不理想。即使提供形态分析器，将语素简单排序后输入标准 BERT 架构，也难以有效捕捉形态组合性并表达相对于词的句法规律。我们通过提出一种简单而有效的双层 BERT 架构来应对这些挑战，该架构利用形态分析器并显式表示形态组合性。尽管 BERT 取得了成功，但其大多数评估都是在高资源语言上进行的，这掩盖了它在低资源语言上的适用性。我们在低资源且形态丰富的卢旺达语上评估所提出的方法，并将提出的模型架构命名为 KinyaBERT。一组可靠的实验结果表明，在命名实体识别任务中，KinyaBERT 的 F1 分数比强基线高出 2%；在机器翻译版 GLUE 基准的平均分数上高出 4.3%。KinyaBERT 的微调具有更好的收敛性，并且即使存在翻译噪声，也能在多个任务上取得更稳健的结果。
英文摘要：
Pre-trained language models such as BERT have been successful at tackling many natural language processing tasks. However, the unsupervised sub-word tokenization methods commonly used in these models (e.g., byte-pair encoding - BPE) are sub-optimal at handling morphologically rich languages. Even given a morphological analyzer, naive sequencing of morphemes into a standard BERT architecture is inefficient at capturing morphological compositionality and expressing word-relative syntactic regularities. We address these challenges by proposing a simple yet effective two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphological compositionality. Despite the success of BERT, most of its evaluations have been conducted on high-resource languages, obscuring its applicability on low-resource languages. We evaluate our proposed method on the low-resource morphologically rich Kinyarwanda language, naming the proposed model architecture KinyaBERT. A robust set of experimental results reveal that KinyaBERT outperforms solid baselines by 2% in F1 score on a named entity recognition task and by 4.3% in average score of a machine-translated GLUE benchmark. KinyaBERT fine-tuning has better convergence and achieves more robust results on multiple tasks even in the presence of translation noise.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
