arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.02858cs.DBcs.DLcs.DS
收藏

降维聚类数据与有序划分及逐步增维索引

Dimensionality Reduced Clustered Data and Order Partition and Stepwise Dimensionality Increasing Indices

Alexander Thomasian

AI总结:

本文回顾了针对高维卫星图像相似性搜索的索引构建研究,通过数据聚类、SVD降维及Ordered Partition和SDI索引技术,显著降低了k-NN查询处理成本并提升了效率。

摘要:

IBM T. J. Watson Research Center一项NASA资助项目的目标之一是为相似性搜索卫星图像构建索引,这些图像由高维特征图像纹理向量表征。本文回顾了我们在数据聚类、通过Singular Value Decomposition - SVD(奇异值分解)进行降维以及索引构建方面的努力,旨在构建更小的索引并实现更高效的k-Nearest Neighbor - k-NN(k-最近邻)查询处理。对于不断增长的图像数量,基于扫描所有图像特征向量的k-NN查询显然成本过高。鉴于普遍存在的多维R-tree索引及其扩展在维度上的可扩展性有限,它们并非可行的选项。通过在降维聚类上构建内存驻留的Ordered Partition(有序划分)索引,进一步降低了处理k-NN查询的成本。在大学环境中进行的后续研究包括:(1) 扩展Clustered SVD(聚类SVD),通过发出适当的成本更低的范围查询来产生精确的k-NN查询;(2) Stepwise Dimensionality Increasing - SDI(逐步增维)索引的性能优于其他已知索引;(3) 选择最佳维度数量以降低查询处理成本;(4) 两种使OP-trees持久化并可作为单文件访问加载的方法。

英文摘要:

One of the goals of NASA funded project at IBM T. J. Watson Research Center was to build an index for similarity searching satellite images, which were characterized by high-dimensional feature image texture vectors. Reviewed is our effort on data clustering, dimensionality reduction via Singular Value Decomposition - SVD and indexing to build a smaller index and more efficient k-Nearest Neighbor - k-NN query processing for similarity search. k-NN queries based on scanning of the feature vectors of all images is obviously too costly for ever-increasing number of images. The ubiquitous multidimensional R-tree index and its extensions were not an option given their limited scalability dimension-wise. The cost of processing k-NN queries was further reduced by building memory resident Ordered Partition indices on dimensionality reduced clusters. Further research in a university setting included the following: (1) Clustered SVD was extended to yield exact k-NN queries by issuing appropriate less costly range queries, (2) Stepwise Dimensionality Increasing - SDI index outperformed other known indices, (3) selection of optimal number of dimensions to reduce query processing cost, (4) two methods to make the OP-trees persistent and loadable as a single file access.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑