arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.07904cs.CL
收藏

HashFormers:迈向词汇独立的预训练Transformers

HashFormers: Towards Vocabulary-independent Pre-trained Transformers

University of Sheffield(谢菲尔德大学)

Huiyin Xue, Nikolaos Aletras

摘要:

基于transformer的预训练语言模型是词汇相关的,默认情况下将每个标记映射到其对应的嵌入。这种一对一的映射导致嵌入矩阵占用大量内存(即数百万个参数),并随着词汇表的大小线性增长。以前的工作对设备上的Transformers器动态生成令牌嵌入的飞行,而不嵌入矩阵使用位置敏感的哈希形态信息。这些嵌入随后被送入Transformer层进行文本分类。然而,这些方法不是预先训练的。受这一系列工作的启发,我们提出了HashFormers,这是一个新的独立于词汇表的预训练Transformers家族,它支持无限的词汇表(即语料库中所有可能的标记),给定一个小得多的固定大小的嵌入矩阵。我们通过首先引入计算成本低的哈希函数来实现这一点,该函数将单个令牌集中到嵌入中。我们还提出了三种完全不需要嵌入矩阵的变体,进一步降低了内存需求。我们的经验表明,与标准的预训练Transformers相比,HashFormers的内存效率更高,同时在多个文本分类任务上进行微调时,可以实现相当的预测性能。例如,我们最有效的HashFormer变体的性能下降可以忽略不计(在GLUE上为0.4\%),仅使用99.1K参数来表示嵌入,而最先进的模型的参数为12.3- 38 M。

英文摘要:

Transformer-based pre-trained language models are vocabulary-dependent, mapping by default each token to its corresponding embedding. This one-to-one mapping results into embedding matrices that occupy a lot of memory (i.e. millions of parameters) and grow linearly with the size of the vocabulary. Previous work on on-device transformers dynamically generate token embeddings on-the-fly without embedding matrices using locality-sensitive hashing over morphological information. These embeddings are subsequently fed into transformer layers for text classification. However, these methods are not pre-trained. Inspired by this line of work, we propose HashFormers, a new family of vocabulary-independent pre-trained transformers that support an unlimited vocabulary (i.e. all possible tokens in a corpus) given a substantially smaller fixed-sized embedding matrix. We achieve this by first introducing computationally cheap hashing functions that bucket together individual tokens to embeddings. We also propose three variants that do not require an embedding matrix at all, further reducing the memory requirements. We empirically demonstrate that HashFormers are more memory efficient compared to standard pre-trained transformers while achieving comparable predictive performance when fine-tuned on multiple text classification tasks. For example, our most efficient HashFormer variant has a negligible performance degradation (0.4\% on GLUE) using only 99.1K parameters for representing the embeddings compared to 12.3-38M parameters of state-of-the-art models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑