arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1602.02215cs.CL
收藏

Swivel:通过注意缺少的内容来改进嵌入

Swivel: Improving Embeddings by Noticing What's Missing

Google Inc.(谷歌公司)

Noam Shazeer, Ryan Doherty, Colin Evans, Chris Waterson

摘要:

我们提出了子矩阵式向量嵌入学习器(Swivel),一种从特征共生矩阵生成低维特征嵌入的方法。Swivel通过随机梯度下降执行逐点互信息矩阵的近似因式分解。它使用分段损失,并对未观察到的同现进行特殊处理,从而利用矩阵中的所有信息。虽然这需要与整个矩阵的大小成比例的计算,但我们使用向量化乘法来同时处理数千个行和列,以计算数百万个预测值。此外,我们将矩阵划分为分片,以便在许多节点上并行计算。这种方法比只考虑观察到的同现的方法更准确地嵌入,并且可以扩展到比采样方法更大的语料库。

英文摘要:

We present Submatrix-wise Vector Embedding Learner (Swivel), a method for generating low-dimensional feature embeddings from a feature co-occurrence matrix. Swivel performs approximate factorization of the point-wise mutual information matrix via stochastic gradient descent. It uses a piecewise loss with special handling for unobserved co-occurrences, and thus makes use of all the information in the matrix. While this requires computation proportional to the size of the entire matrix, we make use of vectorized multiplication to process thousands of rows and columns at once to compute millions of predicted values. Furthermore, we partition the matrix into shards in order to parallelize the computation across many nodes. This approach results in more accurate embeddings than can be achieved with methods that consider only observed co-occurrences, and can scale to much larger corpora than can be handled with sampling methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑