arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.09255cs.CL
收藏

spaCy中的多哈希嵌入

Multi hash embeddings in spaCy

ExplosionAI GmbH(ExplosionAI有限责任公司)

Lester James Miranda, Ákos Kádár, Adriane Boyd, Sofie Van Landeghem, Anders Søgaard, Matthew Honnibal

摘要:

符号的分布式表示是当今机器学习系统的关键技术之一,在现代自然语言处理中起着举足轻重的作用。传统的词嵌入将单独的向量与每个词相关联。虽然这种方法很简单,而且性能也很好,但它需要大量内存来表示大型词汇表。为了减少内存占用,spaCy中的默认嵌入层是哈希嵌入层。它是传统嵌入的随机近似,为大量单词提供唯一的向量,而不显式地为每个单词存储单独的向量。为了能够计算已知和未知单词的有意义的表示,哈希嵌入将每个单词表示为规范化单词形式,子单词信息和单词形状的摘要。这些特征一起产生了一个词的多重嵌入。在这份技术报告中,我们列出了一些历史,并详细介绍了spaCy中的嵌入方法。其次,我们批判性地评估了哈希嵌入架构与多个嵌入命名实体识别数据集从各种领域和语言。实验验证了spaCy嵌入器背后的大多数关键设计选择,但我们也发现了一些令人惊讶的结果。

英文摘要:

The distributed representation of symbols is one of the key technologies in machine learning systems today, playing a pivotal role in modern natural language processing. Traditional word embeddings associate a separate vector with each word. While this approach is simple and leads to good performance, it requires a lot of memory for representing a large vocabulary. To reduce the memory footprint, the default embedding layer in spaCy is a hash embeddings layer. It is a stochastic approximation of traditional embeddings that provides unique vectors for a large number of words without explicitly storing a separate vector for each of them. To be able to compute meaningful representations for both known and unknown words, hash embeddings represent each word as a summary of the normalized word form, subword information and word shape. Together, these features produce a multi-embedding of a word. In this technical report we lay out a bit of history and introduce the embedding methods in spaCy in detail. Second, we critically evaluate the hash embedding architecture with multi-embeddings on Named Entity Recognition datasets from a variety of domains and languages. The experiments validate most key design choices behind spaCy's embedders, but we also uncover a few surprising results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑