arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.14561cs.CL
收藏

如何计算词的概率

How to Compute the Probability of a Word

Tiago Pimentel, Clara Meister

AI总结:

本文指出近期语言学研究中计算词概率的常见错误,推导了基于子词的正确计算方法,并揭示了GPT系列等词首标记分词器导致的问题,证明修正该错误会影响句子理解与词汇优化分析的测量结果。

摘要:

语言模型(LMs)估计自然语言中字符串上的概率分布;这些分布对于计算语言学中的困惑度和惊异度至关重要。虽然我们通常关注测量词的这些值,但大多数LMs在子词上操作。尽管看似简单,但已知一个单元上的概率去准确计算另一个单元上的概率需要谨慎。事实上,我们在此表明,许多近期语言学研究一直错误地计算这些值。本文推导了计算词概率的正确方法,强调了依赖使用词首(bow)标记分词器(例如GPT系列)的语言模型时的问题。在经验上,我们表明修正概率计算中的普遍错误会影响句子理解和词汇优化分析中的测量结果。

英文摘要:

Language models (LMs) estimate a probability distribution over strings in a natural language; these distributions are crucial for computing perplexity and surprisal in linguistics research. While we are usually concerned with measuring these values for words, most LMs operate over subwords. Despite seemingly straightforward, accurately computing probabilities over one unit given probabilities over the other requires care. Indeed, we show here that many recent linguistic studies have been incorrectly computing these values. This paper derives the correct methods for computing word probabilities, highlighting issues when relying on language models that use beginning-of-word (bow)-marking tokenisers, e.g., the GPT family. Empirically, we show that correcting the widespread bug in probability computations affects measured outcomes in sentence comprehension and lexical optimisation analyses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑