arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 0905.2141cs.DS
收藏

基于索引的相似性搜索问题中的模糊性问题

Curse of Dimensionality in the Application of Pivot-based Indexes to the Similarity Search Problem

Ilya Volnyansky

摘要:

在这项工作中,我们研究的有效性,所谓的灾难的维数索引的数据库相似性搜索。我们进行渐近分析,与测试模型的基础上的一系列度量空间$(Ω_d)$,我们挑选数据集$X_d$在一个i.i.d.时尚.我们将下标$d$称为空间$Ω_d$的维数(例如,对于$\mathbb{R}^d$,维数只是通常的维数),并且我们允许数据集$n=n_d$的大小使得$d$在$n$中是超对数但次多项式。我们研究的渐近性能的基于索引的索引方案的枢轴的数量是$o(n/d)$。我们选择了相对简单的相似性搜索成本模型,其中我们将每个距离计算作为单个计算,而忽略其余部分。我们证明,如果空间$Ω_d$表现出(相当常见的)浓度测量现象的性能相似性搜索使用这样的索引是渐近线性的$n$。也就是说,对于足够大的$d$,使用这样的索引和执行根本没有索引的搜索之间的差异是可以忽略的。因此,我们确认在这种情况下的维数灾难。

英文摘要:

In this work we study the validity of the so-called curse of dimensionality for indexing of databases for similarity search. We perform an asymptotic analysis, with a test model based on a sequence of metric spaces $(Ω_d)$ from which we pick datasets $X_d$ in an i.i.d. fashion. We call the subscript $d$ the dimension of the space $Ω_d$ (e.g. for $\mathbb{R}^d$ the dimension is just the usual one) and we allow the size of the dataset $n=n_d$ to be such that $d$ is superlogarithmic but subpolynomial in $n$. We study the asymptotic performance of pivot-based indexing schemes where the number of pivots is $o(n/d)$. We pick the relatively simple cost model of similarity search where we count each distance calculation as a single computation and disregard the rest. We demonstrate that if the spaces $Ω_d$ exhibit the (fairly common) concentration of measure phenomenon the performance of similarity search using such indexes is asymptotically linear in $n$. That is for large enough $d$ the difference between using such an index and performing a search without an index at all is negligeable. Thus we confirm the curse of dimensionality in this setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑