arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.07923cs.CL
收藏

词边界信息对编码器语言模型并无用处

Word Boundary Information Isn't Useful for Encoder Language Models

University of Sheffield(谢菲尔德大学);University of Bath(巴斯大学)

Edward Gow-Smith, Dylan Phelps, Harish Tayyar Madabushi, Carolina Scarton, Aline Villavicencio

AI总结:

本文通过预训练29个模型,在多种任务上验证了去除词边界信息不会损害Transformer编码器性能,表明词边界信息并非必需。

摘要:

所有现有的基于子词分词算法的Transformer NLP方法,都通过使用构成词元一部分的特殊空格符号(如##或_)来编码空白(词边界信息)。这些符号已被证明会a)导致分词结果的形态有效性降低,以及b)造成大量词汇冗余。因此,去除这些符号已被证明在预训练-微调范式中,对Transformer编码器处理形态复杂的词有有益效果。在本工作中,我们探讨词边界信息对这些模型是否完全有用。具体而言,我们在四种不同的训练规模下训练Transformer编码器,并研究了几种包含词边界信息的替代方法,在跨不同领域和问题设置的一系列任务上进行了评估:GLUE(用于句子级分类)、NER(用于词元级分类),以及两个涉及复杂词的分类数据集(Superbizarre和FLOTA)。总体而言,通过包括预训练29个模型的广泛实验设置,我们发现我们的替代方法没有带来实质性改进,这表明修改分词器以去除词边界信息并不会导致有用信息的丢失。

英文摘要:

All existing transformer-based approaches to NLP using subword tokenisation algorithms encode whitespace (word boundary information) through the use of special space symbols (such as \#\# or \_) forming part of tokens. These symbols have been shown to a) lead to reduced morphological validity of tokenisations, and b) give substantial vocabulary redundancy. As such, removing these symbols has been shown to have a beneficial effect on the processing of morphologically complex words for transformer encoders in the pretrain-finetune paradigm. In this work, we explore whether word boundary information is at all useful to such models. In particular, we train transformer encoders across four different training scales, and investigate several alternative approaches to including word boundary information, evaluating on a range of tasks across different domains and problem set-ups: GLUE (for sentence-level classification), NER (for token-level classification), and two classification datasets involving complex words (Superbizarre and FLOTA). Overall, through an extensive experimental setup that includes the pre-training of 29 models, we find no substantial improvements from our alternative approaches, suggesting that modifying tokenisers to remove word boundary information isn't leading to a loss of useful information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑