《全唐诗》分词研究
Onto Word Segmentation of the Complete Tang Poems
摘要：
我们的目标是对《全唐诗》中的词进行切分。虽然可以在不进行全面分词的情况下对CTP进行一些研究，但为了进行高级研究，我们必须向前推进CTP的词级分析。2018年11月，当我们为DH 2019（ADHO）提交稿件时，我们只收集了2433首由训练有素的专家分割的诗歌，并使用分割的诗歌来评估考虑中国诗歌领域知识的分割器。我们基于CTP诗歌（不包括2433首诗，这是专门用于测试）和领域知识训练汉字之间的逐点互信息（PMI）。切分器依靠PMI信息，对测试诗中85.7%的词进行了切分。然而，我们可以完全正确地分割一首诗的概率只有17.8%。当我们在DH 2019上展示我们的工作时，我们已经注释了20000多首诗。有了大量的数据，我们能够将biLSTM模型应用于这个分词任务，我们在20%以上的时间内完全正确地分割了一首诗。相比之下，人类注释者在大约40%的时间内完全同意他们的注释。
英文摘要：
We aim at segmenting words in the Complete Tang Poems (CTP). Although it is possible to do some research about CTP without doing full-scale word segmentation, we must move forward to word-level analysis of CTP for conducting advanced research topics. In November 2018 when we submitted the manuscript for DH 2019 (ADHO), we collected only 2433 poems that were segmented by trained experts, and used the segmented poems to evaluate the segmenter that considered domain knowledge of Chinese poetry. We trained pointwise mutual information (PMI) between Chinese characters based on the CTP poems (excluding the 2433 poems, which were used exclusively only for testing) and the domain knowledge. The segmenter relied on the PMI information to the recover 85.7% of words in the test poems. We could segment a poem completely correct only 17.8% of the time, however. When we presented our work at DH 2019, we have annotated more than 20000 poems. With a much larger amount of data, we were able to apply biLSTM models for this word segmentation task, and we segmented a poem completely correct above 20% of the time. In contrast, human annotators completely agreed on their annotations about 40% of the time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
