Ultra-Quantisation:通过1.58位编码实现高效嵌入搜索
Ultra-Quantisation: Efficient Embedding Search via 1.58-bit Encodings
AI总结:
本文提出Ultra-Quantisation方法,将高维浮点嵌入向量替换为元素取自{-1,0,1}的1.58位编码向量,通过凸多面体实现激进量化,在显著节省空间和计算成本的同时保持相似性测量精度。
摘要:
许多现代搜索领域包含由神经网络生成的浮点数高维向量,即嵌入(embeddings)。典型的嵌入维度从数百到数千不等,这使得嵌入的大小和比较速度成为重要问题。量化是一类用更小的表示(例如短整数)替换浮点值的机制,通过近似嵌入空间换取更小的数据表示和更快的比较函数。在此,我们将这一思想推向近乎极端:展示如何将任意精度的浮点值向量替换为元素取自集合{-1,0,1}的向量。这在空间和度量评估成本上带来显著节省,同时保持相似性测量的强相关性。该方法通过存在于高维空间中的一类凸多面体(convex polytopes)实现。本文概述了这些对象的描述,并展示如何将其作为此类激进量化的基础,同时保持令人惊讶的准确度。
英文摘要:
Many modern search domains comprise high-dimensional vectors of floating point numbers derived from neural networks, in the form of embeddings. Typical embeddings range in size from hundreds to thousands of dimensions, making the size of the embeddings, and the speed of comparison, a significant issue. Quantisation is a class of mechanism which replaces the floating point values with a smaller representation, for example a short integer. This gives an approximation of the embedding space in return for a smaller data representation and a faster comparison function. Here we take this idea almost to its extreme: we show how vectors of arbitrary-precision floating point values can be replaced by vectors whose elements are drawn from the set {-1,0,1}. This yields very significant savings in space and metric evaluation cost, while maintaining a strong correlation for similarity measurements. This is achieved by way of a class of convex polytopes which exist in the high-dimensional space. In this article we give an outline description of these objects, and show how they can be used for the basis of such radical quantisation while maintaining a surprising degree of accuracy.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。