arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.04738cs.SEcs.LG
收藏

关于软件工程领域中用于自然语言处理的预训练Transformers的有效性

On the validity of pre-trained transformers for natural language processing in the software engineering domain

University of Goettingen(哥廷根大学);TU Clausthal(克劳斯塔尔工业大学)

Julian von der Mosel, Alexander Trautsch, Steffen Herbold

摘要:

Transformers是当前许多领域中最先进的自然语言处理技术,并且也在软件工程研究中发挥着牵引作用。这些模型是在大量数据上预先训练的,通常来自一般领域。然而,我们对Transformers在软件工程领域内的有效性只有有限的理解,即,这些模型在理解软件工程上下文中的单词和句子方面有多好,以及这如何提高了最先进的水平。在本文中,我们将阐明这个复杂但关键的问题。我们比较了用软件工程数据训练的BERT Transformer模型和基于一般领域数据的Transformers模型在多个维度上的表现:它们的词汇量,它们理解哪些单词缺失的能力,以及它们在分类任务中的表现。我们的研究结果表明,对于需要理解软件工程上下文的任务,使用软件工程数据进行预训练是有价值的,而一般的领域模型对于一般的语言理解是足够的,也在软件工程领域内。

英文摘要:

Transformers are the current state-of-the-art of natural language processing in many domains and are using traction within software engineering research as well. Such models are pre-trained on large amounts of data, usually from the general domain. However, we only have a limited understanding regarding the validity of transformers within the software engineering domain, i.e., how good such models are at understanding words and sentences within a software engineering context and how this improves the state-of-the-art. Within this article, we shed light on this complex, but crucial issue. We compare BERT transformer models trained with software engineering data with transformers based on general domain data in multiple dimensions: their vocabulary, their ability to understand which words are missing, and their performance in classification tasks. Our results show that for tasks that require understanding of the software engineering context, pre-training with software engineering data is valuable, while general domain models are sufficient for general language understanding, also within the software engineering domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑