arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.17271cs.CL
收藏

理解输入词元字符在语言模型中的作用:信息损失如何影响性能?

Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance?

University of Sheffield(谢菲尔德大学)

Ahmed Alajrami, Katerina Margatina, Nikolaos Aletras

AI总结:

本文研究输入词元字符信息损失对预训练语言模型性能的影响,发现即使每个词元仅保留一个字符进行预训练,模型在SuperGLUE和GLUE等基准上仍能保持较高性能。

摘要:

理解预训练语言模型(PLMs)如何以及学到了哪些语言知识,是自然语言处理中的一个开放性挑战。以往工作主要关注它们是否捕获了语义和句法信息,以及数据或预训练目标如何影响其性能。然而,据我们所知,尚无先前工作专门研究输入词元字符中的信息损失如何影响预训练语言模型的性能。在本研究中,我们通过使用单个词元中很小的字符子集来预训练语言模型,以填补这一空白。令人惊讶的是,我们发现即使在极端设置下进行预训练,即每个词元仅使用一个字符,与全词元模型相比,在标准自然语言理解基准和探测任务中的性能保持率仍然很高。例如,仅使用词元第一个单字符进行预训练的模型,在SuperGLUE和GLUE任务中分别达到全词元模型约90%和77%的性能保持率。

英文摘要:

Understanding how and what pre-trained language models (PLMs) learn about language is an open challenge in natural language processing. Previous work has focused on identifying whether they capture semantic and syntactic information, and how the data or the pre-training objective affects their performance. However, to the best of our knowledge, no previous work has specifically examined how information loss in input token characters affects the performance of PLMs. In this study, we address this gap by pre-training language models using small subsets of characters from individual tokens. Surprisingly, we find that pre-training even under extreme settings, i.e. using only one character of each token, the performance retention in standard NLU benchmarks and probing tasks compared to full-token models is high. For instance, a model pre-trained only on single first characters from tokens achieves performance retention of approximately $90$\% and $77$\% of the full-token model in SuperGLUE and GLUE tasks, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑