arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.14614cs.CL
收藏

针对分词与无噪声信道的两个反例

Two Counterexamples to Tokenization and the Noiseless Channel

Tokyo Institute of Technology(东京工业大学);ETH Zurich(苏黎世联邦理工学院)

Marco Cognetta, Vilém Zouhar, Sangwhan Moon, Naoaki Okazaki

AI总结:

本文构造两种 BPE 分词反例,证明 Rényi 效率可被任意提高却伴随下游性能下降,从而揭示其作为分词器内在预测指标的局限。

摘要:

在《Tokenization and the Noiseless Channel》(Zouhar 等人,2023a)中,Rényi 效率被提出作为评估分词器的一种内在机制:对于 NLP 任务,应选择能使 unigram 分布获得最高 Rényi 效率的分词器。因此,Rényi 效率被视为下游性能的预测指标(例如预测机器翻译任务的 BLEU),从而无需使用不同分词器训练多个模型这一昂贵步骤。尽管该指标有用,但其预测能力并不完美,作者也指出,良好分词方案还具有 Rényi 效率本身无法捕捉的其他性质。我们描述了两种 BPE 分词变体,它们可以任意提高 Rényi 效率,同时降低下游模型性能。这些反例揭示了 Rényi 效率作为内在分词指标失效的情形,从而为构建更准确的预测器提供启示。

英文摘要:

In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase Rényi efficiency while decreasing the downstream model performance. These counterexamples expose cases where Rényi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑