MarkBERT：标记词边界提高汉语BERT
MarkBERT: Marking Word Boundaries Improves Chinese BERT
摘要：
我们提出了一个中文的BERT模型称为马克BERT，在这项工作中使用的词信息。现有的基于词的BERT模型以词为基本单位，但由于BERT的词汇量限制，只能覆盖高频词，遇到词汇表外（out-of-vocabulary，OOV）词时又退回到字符级。与现有的作品不同，MarkBERT保持词汇是汉字，并在相邻的单词之间插入边界标记。这种设计使得模型能够以相同的方式处理任何单词，无论它们是否是OOV单词。此外，我们的模型还有两个额外的好处：第一，它可以方便地在标记上添加单词级学习目标，这是对传统字符和单词级预训练任务的补充;第二，它可以通过用特定于POS标签的标记替换通用标记来轻松地整合更丰富的语义，例如单词的POS标签。通过简单的标记插入，MarkBERT可以提高各种下游任务的性能，包括语言理解和序列标记。\footnote{All the codes and models will be made publicly available at \url{https://github.com/daiyongya/markbert}}
英文摘要：
We present a Chinese BERT model dubbed MarkBERT that uses word information in this work. Existing word-based BERT models regard words as basic units, however, due to the vocabulary limit of BERT, they only cover high-frequency words and fall back to character level when encountering out-of-vocabulary (OOV) words. Different from existing works, MarkBERT keeps the vocabulary being Chinese characters and inserts boundary markers between contiguous words. Such design enables the model to handle any words in the same way, no matter they are OOV words or not. Besides, our model has two additional benefits: first, it is convenient to add word-level learning objectives over markers, which is complementary to traditional character and sentence-level pretraining tasks; second, it can easily incorporate richer semantics such as POS tags of words by replacing generic markers with POS tag-specific markers. With the simple markers insertion, MarkBERT can improve the performances of various downstream tasks including language understanding and sequence labeling. \footnote{All the codes and models will be made publicly available at \url{https://github.com/daiyongya/markbert}}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
