arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.09657cs.CLcs.AI
收藏

PySBD: Pragmatic Sentence Boundary Disambiguation

Episource LLC;Allen Institute for Artificial Intelligence(艾伦人工智能研究所)

Nipun Sadvilkar, Mark Neumann

英文摘要:

In this paper, we present a rule-based sentence boundary disambiguation Python package that works out-of-the-box for 22 languages. We aim to provide a realistic segmenter which can provide logical sentences even when the format and domain of the input text is unknown. In our work, we adapt the Golden Rules Set (a language-specific set of sentence boundary exemplars) originally implemented as a ruby gem - pragmatic_segmenter - which we ported to Python with additional improvements and functionality. PySBD passes 97.92% of the Golden Rule Set exemplars for English, an improvement of 25% over the next best open-source Python tool.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑