使用BOS和EOS标签组合的句子识别
Sentence Identification with BOS and EOS Label Combinations
摘要：
句子是许多NLP应用程序中的基本单位。句子分割被广泛用作第一个预处理任务，其中输入文本被分割成连续的句子，考虑句子的结尾（EOS）作为它们的边界。这个任务的制定依赖于一个强有力的假设，即输入文本仅由句子组成，或者我们称之为句子单位（SU）。然而，现实世界的文本往往包含非实体单元（NSU），如元数据，句子片段，非语言标记等，这是不合理的或不希望被视为一个SU的一部分。为了解决这个问题，我们制定了一个新的任务，句子识别，其目标是确定SU，同时排除在给定的文本中的NSU。为了进行句子识别，我们提出了一种简单而有效的方法，结合句首（BOS）和EOS标签，以确定最可能的SU和NSU的基础上动态规划。为了评估这一任务，我们设计了一个自动的，独立于语言的程序转换成句子识别基准的通用词典语料库。最后，我们在句子识别任务上的实验表明，我们提出的方法通常优于只利用EOS标签的句子分割基线。
英文摘要：
The sentence is a fundamental unit in many NLP applications. Sentence segmentation is widely used as the first preprocessing task, where an input text is split into consecutive sentences considering the end of the sentence (EOS) as their boundaries. This task formulation relies on a strong assumption that the input text consists only of sentences, or what we call the sentential units (SUs). However, real-world texts often contain non-sentential units (NSUs) such as metadata, sentence fragments, nonlinguistic markers, etc. which are unreasonable or undesirable to be treated as a part of an SU. To tackle this issue, we formulate a novel task of sentence identification, where the goal is to identify SUs while excluding NSUs in a given text. To conduct sentence identification, we propose a simple yet effective method which combines the beginning of the sentence (BOS) and EOS labels to determine the most probable SUs and NSUs based on dynamic programming. To evaluate this task, we design an automatic, language-independent procedure to convert the Universal Dependencies corpora into sentence identification benchmarks. Finally, our experiments on the sentence identification task demonstrate that our proposed method generally outperforms sentence segmentation baselines which only utilize EOS labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
