arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1709.03708cs.CVcs.MM
收藏

PQk-means:乘积量化码的十亿级聚类

PQk-means: Billion-scale Clustering for Product-quantized Codes

National Institute of Informatics(国立信息学研究所);DWANGO Co., Ltd.(DWANGO株式会社);The University of Tokyo(东京大学)

Yusuke Matsui, Keisuke Ogaki, Toshihiko Yamasaki, Kiyoharu Aizawa

摘要:

数据聚类是数据分析中的一个基本操作。对于处理大规模数据,标准的k-means聚类方法不仅速度慢,而且内存效率低。我们提出了一种有效的聚类方法,十亿规模的特征向量,称为PQk-均值。通过首先将输入向量压缩成短的乘积量化(PQ)代码,PQk-均值实现了快速和内存高效的聚类,即使是高维向量。与k均值类似,PQk均值重复分配和更新步骤,这两个步骤都可以在PQ码域中执行。实验结果表明,即使是短长度(32位)的PQ码可以产生竞争力的结果相比,k-means。这一结果对于内存受限环境下的聚类具有实际意义。使用所提出的PQk-means方案,在14小时内实现了10亿个K = 10^5的128 D SIFT特征的聚类,在单台计算机上仅使用32 GB的内存消耗。

英文摘要:

Data clustering is a fundamental operation in data analysis. For handling large-scale data, the standard k-means clustering method is not only slow, but also memory-inefficient. We propose an efficient clustering method for billion-scale feature vectors, called PQk-means. By first compressing input vectors into short product-quantized (PQ) codes, PQk-means achieves fast and memory-efficient clustering, even for high-dimensional vectors. Similar to k-means, PQk-means repeats the assignment and update steps, both of which can be performed in the PQ-code domain. Experimental results show that even short-length (32 bit) PQ-codes can produce competitive results compared with k-means. This result is of practical importance for clustering in memory-restricted environments. Using the proposed PQk-means scheme, the clustering of one billion 128D SIFT features with K = 10^5 is achieved within 14 hours, using just 32 GB of memory consumption on a single computer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑