arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.09255cs.CL
收藏

基于Transformer语言模型和n-gram困惑度的文本向量化

Text vectorization via transformer-based language models and n-gram perplexities

University of Belgrade(贝尔格莱德大学)

Mihailo Škorić

AI总结:

针对标量困惑度易受单个低概率词元影响、丢失文本内部概率分布信息的问题,本研究提出基于n-gram困惑度计算文本向量的算法,通过组合各词元相对困惑度生成文本表示。

摘要:

由于文本的概率(以及困惑度)是基于各个词元的概率乘积计算的,可能出现单个低概率词元大幅降低原本高概率输入的概率(即提升困惑度)的情况,而该词元可能只是简单的排版错误。此外,困惑度是针对整个输入的标量值,计算过程中会丢失输入内部概率分布的信息(一篇有一个低概率词元的质量较好的文本,与另一篇每个词元概率均等的文本,可能具有相同的困惑度值),长文本中尤为明显。作为标量困惑度的替代方案,本研究提出了一种简单算法,用于基于输入内部的n-gram困惑度计算向量值。这类表示方法考虑了上述问题,它不计算单一值,而是计算每个文本词元的相对困惑度,并将这些值组合成一个代表输入的单一向量。

英文摘要:

As the probability (and thus perplexity) of a text is calculated based on the product of the probabilities of individual tokens, it may happen that one unlikely token significantly reduces the probability (i.e., increase the perplexity) of some otherwise highly probable input, while potentially representing a simple typographical error. Also, given that perplexity is a scalar value that refers to the entire input, information about the probability distribution within it is lost in the calculation (a relatively good text that has one unlikely token and another text in which each token is equally likely they can have the same perplexity value), especially for longer texts. As an alternative to scalar perplexity this research proposes a simple algorithm used to calculate vector values based on n-gram perplexities within the input. Such representations consider the previously mentioned aspects, and instead of a unique value, the relative perplexity of each text token is calculated, and these values are combined into a single vector representing the input.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑