arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1611.06172cs.DCstat.ML
收藏

在多核和众核架构中实现Word2Vec的并行化

Parallelizing Word2Vec in Multi-Core and Many-Core Architectures

Shihao Ji, Nadathur Satish, Sheng Li, Pradeep Dubey

摘要:

Word 2 vec是一种广泛使用的算法,用于提取单词的低维向量表示。包括Mikolov等人的那些算法在内的最先进的算法已经被并行化用于多核CPU架构,但是基于具有“Hogwild”更新的向量-向量操作,其是存储器带宽密集型的并且不能有效地使用计算资源。在本文中,我们提出了“HogBatch”通过提高重用的各种数据结构在算法中,通过使用minimizing和负样本共享,从而使我们能够表示使用矩阵乘法运算的问题。我们还探讨了不同的技术来分配word 2 vec计算在计算集群中的节点,并表现出良好的强大的可扩展性高达32个节点。新算法特别适用于现代多核/众核架构,尤其是英特尔最新的Knights Landing处理器,并允许我们在内核和节点之间近似线性地扩展计算,每秒处理数亿个单词,这是我们所知的最快的word 2 vec实现。

英文摘要:

Word2vec is a widely used algorithm for extracting low-dimensional vector representations of words. State-of-the-art algorithms including those by Mikolov et al. have been parallelized for multi-core CPU architectures, but are based on vector-vector operations with "Hogwild" updates that are memory-bandwidth intensive and do not efficiently use computational resources. In this paper, we propose "HogBatch" by improving reuse of various data structures in the algorithm through the use of minibatching and negative sample sharing, hence allowing us to express the problem using matrix multiply operations. We also explore different techniques to distribute word2vec computation across nodes in a compute cluster, and demonstrate good strong scalability up to 32 nodes. The new algorithm is particularly suitable for modern multi-core/many-core architectures, especially Intel's latest Knights Landing processors, and allows us to scale up the computation near linearly across cores and nodes, and process hundreds of millions of words per second, which is the fastest word2vec implementation to the best of our knowledge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑