arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09949cs.CLcs.LG
收藏

面向序列压缩的多词分词方法

Multi-word Tokenization for Sequence Compression

University of Sussex(萨塞克斯大学)

Leonidas Gee, Leonardo Rigutini, Marco Ernandes, Andrea Zugarini

AI总结:

本文提出多词分词器MWT,将频繁多词表达编码为单个token,在固定序列长度下提升覆盖率和性能,并通过缩短序列实现更快推理,支持早期截断加速。

摘要:

大型语言模型已被证明在建模各种任务方面非常成功。然而,这伴随着高昂的计算成本,阻碍了更广泛的工业应用。在本文中,我们提出了MWT:一种多词分词器,它超越词边界,将频繁出现的多词表达表示为单个token。MWT产生更紧凑、更高效的分词结果,带来两个好处:(1)在固定序列长度预算下,由于对输入数据的覆盖范围更大,性能得以提升;(2)由于能够在性能下降可忽略不计的情况下缩短序列长度,推理速度更快、更轻量。我们的结果表明,MWT在较短的序列长度下更具鲁棒性,从而允许通过早期序列截断实现大幅加速。

英文摘要:

Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑