抛下锚点:在语言建模中对数字进行语义启动
Laying Anchors: Semantically Priming Numerals in Language Modeling
AI总结:
本文提出通过语料库数字分布生成锚点来对数字进行语义启动,以改善预训练语言模型的数值编码能力,并在1到100亿的广泛数字范围上验证了方法的有效性。
摘要:
现成的预训练语言模型已成为众多下游任务自然语言处理流程中的事实标准。然而,这些模型无法正确编码数字,这限制了它们在需要数值理解的任务上的表现。我们引入了在任意语料库中对数字进行语义启动的策略,通过生成由该语料库中数字分布所控制的锚点,从而为这些数字标记建立具有数学基础的表示。我们通过一系列面向域内(已见)和域外(未见)数字的算术任务评估,确立了我们所提技术的优越性。此外,我们将实证评估扩展到从1到100亿的数字,与以往同类研究相比范围显著更广,并且我们证明了所学嵌入在数学基础上的显著改进。
英文摘要:
Off-the-shelf pre-trained language models have become the de facto standard in NLP pipelines for a multitude of downstream tasks. However, the inability of these models to properly encode numerals limits their performance on tasks requiring numeric comprehension. We introduce strategies to semantically prime numerals in any corpus by generating anchors governed by the distribution of numerals in said corpus, thereby enabling mathematically grounded representations of these numeral tokens. We establish the superiority of our proposed techniques through evaluation on a range of numeracy tasks for both in-domain (seen) and out-domain (unseen) numerals. Further, we expand our empirical evaluations to numerals ranging from 1 to 10 billion, a significantly broader range compared to previous studies of the same nature, and we demonstrate significant improvements in the mathematical grounding of our learned embeddings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。