网络高效的分布式大词汇量Word2vec训练系统
Network-Efficient Distributed Word2vec Training System for Large Vocabularies
摘要：
Word2vec是一个流行的算法家族，用于在大型文本语料库上对单词的密集向量表示进行无监督训练。所得到的向量已被证明可以捕获其对应单词之间的语义关系，并且在将许多自然语言处理（NLP）任务减少为对这些向量的数学运算方面显示出了希望。虽然迄今为止word2vec的应用都集中在具有几百万个单词的词汇表上，其中词汇表是同时训练向量的单词集合，但在NLP之外的领域中，新的应用正在出现，词汇表包括数亿个单词。现有的word2vec训练系统对于训练如此大的词汇表是不切实际的，因为它们要么要求所有词汇表单词的向量被存储在单个服务器的存储器中，要么由于大量的网络数据传输而遭受不可接受的训练延迟。在本文中，我们提出了一种新的分布式，并行训练系统，使前所未有的实际训练向量的词汇量有几个100万字的商品服务器上的共享集群，使用远低于现有的解决方案的网络流量。我们在一个基准数据集上对所提出的系统进行了评估，结果表明，相对于非分布式训练，向量的质量不会降低。最后，在几个季度里，该系统已被部署用于将查询与雅虎赞助搜索广告平台Gemini中的广告进行匹配，从而显著改善了业务指标。
英文摘要：
Word2vec is a popular family of algorithms for unsupervised training of dense vector representations of words on large text corpuses. The resulting vectors have been shown to capture semantic relationships among their corresponding words, and have shown promise in reducing a number of natural language processing (NLP) tasks to mathematical operations on these vectors. While heretofore applications of word2vec have centered around vocabularies with a few million words, wherein the vocabulary is the set of words for which vectors are simultaneously trained, novel applications are emerging in areas outside of NLP with vocabularies comprising several 100 million words. Existing word2vec training systems are impractical for training such large vocabularies as they either require that the vectors of all vocabulary words be stored in the memory of a single server or suffer unacceptable training latency due to massive network data transfer. In this paper, we present a novel distributed, parallel training system that enables unprecedented practical training of vectors for vocabularies with several 100 million words on a shared cluster of commodity servers, using far less network traffic than the existing solutions. We evaluate the proposed system on a benchmark dataset, showing that the quality of vectors does not degrade relative to non-distributed training. Finally, for several quarters, the system has been deployed for the purpose of matching queries to ads in Gemini, the sponsored search advertising platform at Yahoo, resulting in significant improvement of business metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
