arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.01273cs.DC
收藏

组块内容不够:基于组块上下文感知的增量压缩相似检测

Chunk Content is not Enough: Chunk-Context Aware Resemblance Detection for Deduplication Delta Compression

Xuming Ye, Xiaoye Xue, Wenlong Tian, Zhiyong Xu, Weijun Xiao, Ruixuan Li

摘要:

随着云存储的日益普及,删除用户之间的重复数据对于服务提供商降低成本变得越来越重要。数据相似性检测是近年来出现的一种检测相似性之间冗余的新技术。它从每个块内容中提取特征,并将相似度高的块作为去除冗余的候选。然而,流行的相似性方法,如“N-变换”和“精细”,只使用块数据的特征提取。对数据块的微小修改可能会严重恶化其相似性检测的能力。在本文中,我们提出了一种新的块上下文感知相似性检测算法,称为CARD,以减轻这个问题。CARD引入了基于BP神经网络的组块上下文感知模型,并采用了基于N个子组块瓦片的初始特征提取策略。它有效地将每个数据块内容的内部结构与上下文信息结合起来进行特征提取,显著降低了数据块微小变化的影响。为了评估其性能,我们实现了CARD原型,并使用真实世界的数据集进行了广泛的实验。结果表明,CARD可以检测到高达75.03%的冗余数据和加速相似性检测操作的5.6到17.8倍的速度相比,国家的最先进的相似性检测方法。

英文摘要:

With the growing popularity of cloud storage, removing duplicated data across users is getting more critical for service providers to reduce costs. Recently, Data resemblance detection is a novel technology to detect redundancy among similarity. It extracts feature from each chunk content and treat chunks with high similarity as candidates for removing redundancy. However, popular resemblance methods such as "N-transform" and "Finesse" use only the chunk data for feature extraction. A minor modification on the data chunk could seriously deteriorate its capability for resemblance detection. In this paper, we proposes a novel chunk-context aware resemblance detection algorithm, called CARD, to mitigate this issue. CARD introduces a BP-Neural network-based chunk-context aware model, and uses N-sub-chunk shingles-based initial feature extraction strategy. It effectively integrates each data chunk content's internal structure with the context information for feature extraction, the impact of small changes in data chunks is significantly reduced. To evaluate its performance, we implement a CARD prototype and conduct extensive experiments using real-world data sets. The results show that CARD can detect up to 75.03% more redundant data and accelerate the resemblance detection operations by 5.6 to 17.8 times faster compared with the state-of-the-art resemblance detection approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑