arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.13225cs.LGcs.AIcs.DCcs.IRstat.ML
收藏

SOLAR:稀疏正交学习和随机嵌入

SOLAR: Sparse Orthogonal Learned and Random Embeddings

Rice University(莱斯大学)

Tharun Medini, Beidi Chen, Anshumali Shrivastava

摘要:

密集嵌入模型通常部署在商业搜索引擎中,其中所有文档向量都是预先计算的,并且利用查询向量执行近邻搜索(NNS)以找到相关文档。然而,索引大量密集向量和执行NNS的瓶颈损害了这些模型的查询时间和准确性。在本文中,我们认为,高维和超稀疏嵌入是一个显着优于密集的低维嵌入的查询效率和准确性。极端稀疏性通过用简单的查找替换NNS来消除对NNS的需要,而其高维性确保即使在稀疏时嵌入也是信息丰富的。然而,学习极高维嵌入会导致模型大小爆炸。为了使训练可行,我们提出了一种分区算法,可以在没有任何通信的情况下跨多个GPU学习这种高维嵌入。这是由我们的新的稀疏,正交,学习和随机(太阳)嵌入的非对称混合物。标签向量是随机的,稀疏的,并且通过设计接近正交,而查询向量是学习的和稀疏的。我们从理论上证明了我们的单边学习方式等价于同时学习查询和标签嵌入。有了这些独特的属性,我们可以成功地训练50万维SOLAR嵌入,用于在三个最大的公共数据集上搜索160万本书和多标签分类。与每个任务的最新基线相比,我们实现了卓越的精确度和召回率,速度快达10倍。

英文摘要:

Dense embedding models are commonly deployed in commercial search engines, wherein all the document vectors are pre-computed, and near-neighbor search (NNS) is performed with the query vector to find relevant documents. However, the bottleneck of indexing a large number of dense vectors and performing an NNS hurts the query time and accuracy of these models. In this paper, we argue that high-dimensional and ultra-sparse embedding is a significantly superior alternative to dense low-dimensional embedding for both query efficiency and accuracy. Extreme sparsity eliminates the need for NNS by replacing them with simple lookups, while its high dimensionality ensures that the embeddings are informative even when sparse. However, learning extremely high dimensional embeddings leads to blow up in the model size. To make the training feasible, we propose a partitioning algorithm that learns such high dimensional embeddings across multiple GPUs without any communication. This is facilitated by our novel asymmetric mixture of Sparse, Orthogonal, Learned and Random (SOLAR) Embeddings. The label vectors are random, sparse, and near-orthogonal by design, while the query vectors are learned and sparse. We theoretically prove that our way of one-sided learning is equivalent to learning both query and label embeddings. With these unique properties, we can successfully train 500K dimensional SOLAR embeddings for the tasks of searching through 1.6M books and multi-label classification on the three largest public datasets. We achieve superior precision and recall compared to the respective state-of-the-art baselines for each of the tasks with up to 10 times faster speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑