arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2002.04723cs.LGcs.CLstat.ML
收藏

Superbloom:Bloom filter满足Transformer

Superbloom: Bloom filter meets Transformer

Google Research(谷歌研究院)

John Anderson, Qingqing Huang, Walid Krichene, Steffen Rendle, Li Zhang

摘要:

我们将自然语言模型中的词块的概念扩展到不透明id上的机器学习任务。这是通过应用哈希函数将每个id映射到更小空间中的多个哈希令牌来实现的,类似于布隆过滤器。我们表明,通过应用多层Transformer这些布隆过滤器,我们能够获得高精度的模型。它们在没有哈希的情况下优于类似大小的模型,并且在很大程度上优于使用具有相同计算预算的softmax样本训练的更大大小的模型。我们的关键观察是,重要的是使用多层Transformer用于Bloom过滤器,以消除散列输入中的模糊性。我们相信这为解决大词汇量的问题提供了一种替代方法。

英文摘要:

We extend the idea of word pieces in natural language models to machine learning tasks on opaque ids. This is achieved by applying hash functions to map each id to multiple hash tokens in a much smaller space, similarly to a Bloom filter. We show that by applying a multi-layer Transformer to these Bloom filter digests, we are able to obtain models with high accuracy. They outperform models of a similar size without hashing and, to a large degree, models of a much larger size trained using sampled softmax with the same computational budget. Our key observation is that it is important to use a multi-layer Transformer for Bloom filter digests to remove ambiguity in the hashed input. We believe this provides an alternative method to solving problems with large vocabulary size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑