arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2208.13486cs.CL
收藏

naab:一个用于波斯语的即插即用型语料库

naab: A ready-to-use plug-and-play corpus for Farsi

Sharif University of Technology(谢里夫理工大学)

Sadra Sabouri, Elnaz Rahmati, Soroush Gooran, Hossein Sameti

AI总结:

针对波斯语等低资源语言在大型语言模型中表现不佳的问题,作者构建了最大规模、即插即用的波斯语清洗语料库naab,并提供未处理版本与预处理工具包,助力提升模型性能并弥合语言资源差距。

摘要:

大型语言模型(LLMs)的兴起改变了众多自然语言处理(NLP)任务,然而其在波斯语等低资源和中等资源语言上的表现仍落后于英语等资源丰富的语言。为了解决这一差距,我们引入了naab,这是目前最大、公开可用、经过清洗且即插即用的波斯语文本语料库。naab包含130GB的数据,由超过2.5亿个段落和150亿个单词组成。该语料库以波斯语单词NAAB(意为“纯净”或“高级”)命名,可通过Hugging Face公开访问,为研究人员提供了用于波斯语NLP任务的宝贵资源。除naab外,我们还提供了naab-raw,即该数据集的未处理版本,以及一个允许用户清洗其自定义语料库的预处理工具包。这些资源赋能NLP研究人员和从业者,特别是那些专注于低资源语言的人,以提升LLMs在其各自领域的性能,并弥合资源丰富语言与资源贫乏语言之间的差距。

英文摘要:

The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word NAAB (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑