一种有效的稀疏集压缩技术
Efficient Compression Technique for Sparse Sets
摘要：
最近的技术进步已经导致在网络上产生大量数据，例如文本、图像、音频和视频。这些数据中的大多数是高维和稀疏的，例如，用于表示文本的词袋表示。通常，在许多应用中需要对相似的数据点进行有效的搜索，如聚类，最近邻搜索，排名和索引。尽管计算能力有了显著的提高，但在这样的数据集上进行简单的蛮力相似性搜索是低效的，有时甚至是不可能的。因此，期望得到保留数据点之间的相似性的压缩表示。在这项工作中，我们认为数据点集和使用Jaccard相似性作为相似性度量。压缩技术通常根据下列参数进行评价：1）压缩所需的随机性，2）压缩所需的时间，3）压缩后数据的维数，以及4）存储压缩数据所需的空间。理想情况下，数据的压缩表示应该保持每对数据点之间的相似性，同时保持压缩所需的时间和随机性尽可能低。我们表明，Pratap和Kulkarni建议的压缩技术也适用于Jaccard相似性。我们提出了同样的理论证明，并补充它与严格的实验合成以及现实世界的数据集。我们还比较了我们的结果与最先进的“最小独立置换”，并表明，我们的压缩算法达到几乎相同的精度，同时显着减少压缩时间和随机性。
英文摘要：
Recent technological advancements have led to the generation of huge amounts of data over the web, such as text, image, audio and video. Most of this data is high dimensional and sparse, for e.g., the bag-of-words representation used for representing text. Often, an efficient search for similar data points needs to be performed in many applications like clustering, nearest neighbour search, ranking and indexing. Even though there have been significant increases in computational power, a simple brute-force similarity-search on such datasets is inefficient and at times impossible. Thus, it is desirable to get a compressed representation which preserves the similarity between data points. In this work, we consider the data points as sets and use Jaccard similarity as the similarity measure. Compression techniques are generally evaluated on the following parameters --1) Randomness required for compression, 2) Time required for compression, 3) Dimension of the data after compression, and 4) Space required to store the compressed data. Ideally, the compressed representation of the data should be such, that the similarity between each pair of data points is preserved, while keeping the time and the randomness required for compression as low as possible. We show that the compression technique suggested by Pratap and Kulkarni also works well for Jaccard similarity. We present a theoretical proof of the same and complement it with rigorous experimentations on synthetic as well as real-world datasets. We also compare our results with the state-of-the-art "min-wise independent permutation", and show that our compression algorithm achieves almost equal accuracy while significantly reducing the compression time and the randomness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
