利用子词表示学习与位置感知自注意力增强信德语词分割
Enhancing Sindhi Word Segmentation using Subword Representation Learning and Position-aware Self-attention
AI总结：
本文提出子词引导神经词分割器SGNWS，结合双向LSTM、位置感知自注意力和条件随机场进行序列标注，在六个信德语数据集上取得最先进的分割性能。
摘要：
信德语词分割因空格省略和插入问题而成为一项具有挑战性的任务。信德语本身进一步增加了这种复杂性：它是草书体，字符具有固有的连接与非连接属性，且与词边界无关。现有的信德语词分割方法依赖设计和组合手工特征。然而，这些方法存在局限，例如难以处理未登录词、对其他语言的鲁棒性有限，以及处理大量噪声或原始文本时效率低下。相比之下，基于神经网络的模型能够自动捕获词边界信息，无需先验知识。本文提出了一种子词引导神经词分割器（SGNWS），将词分割作为序列标注任务来处理。SGNWS 模型通过双向长短期记忆编码器、位置感知自注意力和条件随机场来引入子词表示学习。我们的实验结果表明，SGNWS 模型在六个数据集上的信德语词分割中取得了最先进的性能。
英文摘要：
Sindhi word segmentation is a challenging task due to space omission and insertion issues. The Sindhi language itself adds to this complexity. It's cursive and consists of characters with inherent joining and non-joining properties, independent of word boundaries. Existing Sindhi word segmentation methods rely on designing and combining hand-crafted features. However, these methods have limitations, such as difficulty handling out-of-vocabulary words, limited robustness for other languages, and inefficiency with large amounts of noisy or raw text. Neural network-based models, in contrast, can automatically capture word boundary information without requiring prior knowledge. In this paper, we propose a Subword-Guided Neural Word Segmenter (SGNWS) that addresses word segmentation as a sequence labeling task. The SGNWS model incorporates subword representation learning through a bidirectional long short-term memory encoder, position-aware self-attention, and a conditional random field. Our empirical results demonstrate that the SGNWS model achieves state-of-the-art performance in Sindhi word segmentation on six datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
