一夜之间为梵文构建分词器
Building a Word Segmenter for Sanskrit Overnight
摘要：
梵语中有大量的数字化文本。然而，在这样的文本中的分词任务是具有挑战性的，由于“Sandhi”的问题。在连读变调中，一个句子中的单词经常融合在一起形成一个单一的文本块，其中单词的重复消失，单词边界的声音经历了变化，这也反映在书面文本中。在这里，我们提出了一种使用深度序列到序列（seq2seq）模型的方法，该模型仅将混叠字符串作为输入并预测未混叠字符串。最先进的模型涉及语言学，并且对于输入的词汇和形态分析具有外部依赖性。我们的模型可以“一夜之间”训练出来，并用于生产。尽管采用了知识精益方法，但我们的系统比现有技术更好，比现有技术增加了16.79%。
英文摘要：
There is an abundance of digitised texts available in Sanskrit. However, the word segmentation task in such texts are challenging due to the issue of 'Sandhi'. In Sandhi, words in a sentence often fuse together to form a single chunk of text, where the word delimiter vanishes and sounds at the word boundaries undergo transformations, which is also reflected in the written text. Here, we propose an approach that uses a deep sequence to sequence (seq2seq) model that takes only the sandhied string as the input and predicts the unsandhied string. The state of the art models are linguistically involved and have external dependencies for the lexical and morphological analysis of the input. Our model can be trained "overnight" and be used for production. In spite of the knowledge lean approach, our system preforms better than the current state of the art by gaining a percentage increase of 16.79 % than the current state of the art.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
