arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 0906.0391cs.DS
收藏

基于XML的索引中的抽象性诅咒

Curse of Dimensionality in Pivot-based Indexes

Ilya Volnyansky, Vladimir Pestov

摘要:

我们提供了一个理论验证的灾难的维数在基于索引的数据集的相似性搜索,通过证明,在统计学习的框架中,在高维没有基于索引的索引方案可以基本上优于线性扫描。对一系列数据集进行了渐近性能的研究,这些数据集被建模为在i.i.d.中挑选的样本$X_d$。从度量空间$Ω_d$的方式。我们允许数据集$n=n_d$的大小是这样的,即$d$,“维数”,是超对数的,但在$n$中是次多项式的。允许枢轴数按$o(n/d)$增长。我们选择相似性搜索的限制性最小的成本模型,其中我们将每个距离计算作为单个计算,而忽略其余的计算。我们证明了如果度量集中现象意义下的空间$Ω_d$的内禀维数为$O(d)$,则基于相似性搜索的索引在$n$中的性能是渐近线性的.

英文摘要:

We offer a theoretical validation of the curse of dimensionality in the pivot-based indexing of datasets for similarity search, by proving, in the framework of statistical learning, that in high dimensions no pivot-based indexing scheme can essentially outperform the linear scan. A study of the asymptotic performance of pivot-based indexing schemes is performed on a sequence of datasets modeled as samples $X_d$ picked in i.i.d. fashion from metric spaces $Ω_d$. We allow the size of the dataset $n=n_d$ to be such that $d$, the ``dimension'', is superlogarithmic but subpolynomial in $n$. The number of pivots is allowed to grow as $o(n/d)$. We pick the least restrictive cost model of similarity search where we count each distance calculation as a single computation and disregard the rest. We demonstrate that if the intrinsic dimension of the spaces $Ω_d$ in the sense of concentration of measure phenomenon is $O(d)$, then the performance of similarity search pivot-based indexes is asymptotically linear in $n$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑