跨体裁与标点条件下的阿拉伯语句子分割
Arabic Sentence Segmentation Across Genres and Punctuation Conditions
AI总结：
针对阿拉伯语标点歧义和不一致导致的句子分割难题，构建跨8种体裁的语料库AraSEG，评估LLM、轻量编码器和依存解析器，发现轻量模型在困难设置下优于LLM，且准确分割能显著提升下游依存解析。
摘要：
阿拉伯语的句子分割因标点符号的歧义和不一致而具有挑战性，许多文本缺乏可靠的句子边界标记。现有方法严重依赖标点线索，且通常在格式良好的文本上进行评估，限制了其在真实阿拉伯语环境中的鲁棒性。为解决这一问题，我们引入了AraSEG，一个跨体裁的句子分割语料库，涵盖八种体裁以及广泛的标点和文档结构条件。利用AraSEG，我们在日益具有挑战性的分割设置下评估了LLM、轻量级编码器模型和基于依存解析器的模型。我们的实验表明，在最困难的设置中，轻量级编码器甚至基于依存解析器的模型都优于LLM。我们进一步研究了训练数据规模和体裁多样性的影响，发现性能最终会饱和，且跨体裁泛化仍然具有挑战性。我们还证明了准确的句子分割能显著改善下游的依存解析。我们将公开我们的代码、数据和模型。
英文摘要：
Sentence segmentation in Arabic is challenging due to ambiguous and inconsistent punctuation, with many texts lacking reliable sentence boundary markers. Existing approaches rely heavily on punctuation cues and are typically evaluated on well-formed text, limiting their robustness in realistic Arabic settings. To address this, we introduce AraSEG, a genre-diverse sentence segmentation corpus spanning eight genres and a wide range of punctuation and document structure conditions. Using AraSEG, we evaluate LLMs, lightweight encoder models, and dependency parser-based models under increasingly challenging segmentation settings. Our experiments show that lightweight encoders, and even dependency parser-based models, outperform LLMs under the hardest conditions. We further investigate the effects of training data size and genre diversity, finding that performance eventually saturates and cross-genre generalization remains challenging. We also demonstrate that accurate sentence segmentation substantially improves downstream dependency parsing. We make our code, data, and models publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
