arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.10439cs.SEcs.CL
收藏

用于处理软件工程文档的停用词:它们真的重要吗?

Stop Words for Processing Software Engineering Documents: Do they Matter?

The University of Melbourne(墨尔本大学);Monash University(莫纳什大学)

Yaohou Fan, Chetan Arora, Christoph Treude

AI总结:

本研究针对软件工程领域停用词移除的实用性争议,复现三种相关研究工具,基于1万条Stack Overflow问题构建语料库并识别200个领域停用词,证实领域停用词比通用停用词表更能提升工具性能。

摘要:

停用词被认为不具备预测能力,在自然语言处理任务中常被移除。然而,无信息词汇的定义较为模糊,因此大多数算法使用基于通用知识的停用词表来移除停用词。学术界对于停用词移除的实用性一直存在争议,尤其是在特定领域场景下。在本研究中,我们探讨了软件工程语境下停用词移除的实用性。为此,我们复现了相关工作中的三种软件工程研究工具并开展实验。此外,我们基于10,000条Stack Overflow问题构建了一个软件工程领域相关文本语料库,并使用传统信息论方法识别出200个领域特定停用词。我们的结果表明,与使用通用停用词表相比,使用领域特定停用词显著提升了研究工具的性能,在19项评估指标中有17项表现出更优的性能。

英文摘要:

Stop words, which are considered non-predictive, are often eliminated in natural language processing tasks. However, the definition of uninformative vocabulary is vague, so most algorithms use general knowledge-based stop lists to remove stop words. There is an ongoing debate among academics about the usefulness of stop word elimination, especially in domain-specific settings. In this work, we investigate the usefulness of stop word removal in a software engineering context. To do this, we replicate and experiment with three software engineering research tools from related work. Additionally, we construct a corpus of software engineering domain-related text from 10,000 Stack Overflow questions and identify 200 domain-specific stop words using traditional information-theoretic methods. Our results show that the use of domain-specific stop words significantly improved the performance of research tools compared to the use of a general stop list and that 17 out of 19 evaluation measures showed better performance. Online appendix: https://zenodo.org/record/7865748

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑