词汇接地的子词切分
Lexically Grounded Subword Segmentation
AI总结：
本文提出词汇接地的子词切分方法，结合Morfessor预分词、词嵌入代数推导和子词二元模型高效切分，在8种语言的形态合理性与词性标注上取得一致提升。
摘要：
我们在分词与子词切分方面提出三项创新。第一，我们提出使用基于Morfessor的无监督形态分析作为预分词。第二，我们提出一种代数方法，用于获取以词嵌入空间为接地的子词嵌入。在此基础上，我们设计了一种新的子词切分算法，该算法使用这些嵌入，确保切分过程考虑词汇意义。第三，我们引入一种基于子词二元模型的高效切分算法，该算法可以用词汇感知的切分方法进行初始化，从而在推理时避免使用Morfessor和大型嵌入表。我们使用两个内在指标评估所提出的方法，并在两个下游任务上测量其性能：词性标注和机器翻译。我们的实验表明，在使用语素边界的切分精确率进行评估时，切分在形态学合理性方面有显著提升，并且在8种语言中提高了Rényi效率。尽管所提出的分词方法对自动翻译质量没有大的影响，但我们在更具形态学性质的词性标注任务中观察到一致的性能提升。
英文摘要：
We present three innovations in tokenization and subword segmentation. First, we propose to use unsupervised morphological analysis with Morfessor as pre-tokenization. Second, we present an algebraic method for obtaining subword embeddings grounded in a word embedding space. Based on that, we design a novel subword segmentation algorithm that uses the embeddings, ensuring that the procedure considers lexical meaning. Third, we introduce an efficient segmentation algorithm based on a subword bigram model that can be initialized with the lexically aware segmentation method to avoid using Morfessor and large embedding tables at inference time. We evaluate the proposed approaches using two intrinsic metrics and measure their performance on two downstream tasks: part-of-speech tagging and machine translation. Our experiments show significant improvements in the morphological plausibility of the segmentation when evaluated using segmentation precision on morpheme boundaries and improved Rényi efficiency in 8 languages. Although the proposed tokenization methods do not have a large impact on automatic translation quality, we observe consistent performance gains in the arguably more morphological task of part-of-speech tagging.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
