文本分类可增强领域无关停用词提取
Text Categorization Can Enhance Domain-Agnostic Stopword Extraction
AI总结：
本文结合 MasakhaNEWS 等数据集验证文本分类可有效提取九种非洲语言及法语的领域无关停用词，并提出统计与语言学结合的混合方法。
摘要：
本文研究文本分类在简化自然语言处理（NLP）停用词提取中的作用，重点关注九种非洲语言以及法语。通过利用 MasakhaNEWS、African Stopwords Project 和 MasakhaPOS 数据集，我们的研究结果强调，文本分类能够有效识别领域无关停用词，在大多数受考察语言中检测成功率超过80%。然而，语言差异导致某些语言的检测率较低。有趣的是，我们发现虽然超过40%的停用词在各类新闻类别中通用，但只有不到15%的停用词为单一类别所独有。不常见的停用词能增加文本深度，但其是否被归类为停用词取决于语境。因此，将统计方法与语言学方法相结合可生成全面的停用词表，这凸显了我们混合方法的价值。本研究改进了面向非洲语言的NLP，并强调了文本分类在停用词提取中的重要性。
英文摘要：
This paper investigates the role of text categorization in streamlining stopword extraction in natural language processing (NLP), specifically focusing on nine African languages alongside French. By leveraging the MasakhaNEWS, African Stopwords Project, and MasakhaPOS datasets, our findings emphasize that text categorization effectively identifies domain-agnostic stopwords with over 80% detection success rate for most examined languages. Nevertheless, linguistic variances result in lower detection rates for certain languages. Interestingly, we find that while over 40% of stopwords are common across news categories, less than 15% are unique to a single category. Uncommon stopwords add depth to text but their classification as stopwords depends on context. Therefore combining statistical and linguistic approaches creates comprehensive stopword lists, highlighting the value of our hybrid method. This research enhances NLP for African languages and underscores the importance of text categorization in stopword extraction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
