面向流的“智能压缩”：一种基于有偏库采样的Bloom Filter方法
Towards "Intelligent Compression" in Streams: A Biased Reservoir Sampling based Bloom Filter Approach
摘要：
随着世界范围内存储信息的爆炸式增长，数据密集型计算已成为研究的中心领域。有效管理和处理来自不同来源的大量指数级数据，例如电信呼叫数据记录、在线交易记录等，从如此庞大（数十亿条记录）的数据集中去除冗余，从而提高下游处理的资源和计算效率，这是一个重要的研究领域。流媒体场景中的“智能压缩”或重复数据删除技术，由于数据到达的实时性，从无限数据流中精确识别和消除重复数据是一个更大的挑战。稳定布隆过滤器（Stable Bloom Filter，SBF）在一定程度上解决了这个问题。然而，SBF存在较高的漏报率（FNR）和较慢的收敛速度，从而使其对于FNR容忍度较低的应用程序效率低下。提出了一种新的基于池采样的Bloom Filter（RSBF）数据结构，它将池采样和Bloom Filter的概念相结合，用于数据流中重复项的近似检测.通过详细的理论分析，我们证明了它的误报率（FPR）、漏报率（FNR）和低内存要求下的收敛速度的解析界.我们表明RSBF具有目前最低的误报率和收敛速度，在相同的内存条件下，其性能优于SBF算法（300万条记录）和大约10亿条记录的合成数据集，我们证明了FNR的2倍改进，与SBF相比具有更好的收敛速度，同时表现出相当的FPR。据我们所知，这是首次尝试将存储器采样方法与布隆过滤器集成以用于流场景中的去重。
英文摘要：
With the explosion of information stored world-wide,data intensive computing has become a central area of research.Efficient management and processing of this massively exponential amount of data from diverse sources,such as telecommunication call data records,online transaction records,etc.,has become a necessity.Removing redundancy from such huge(multi-billion records) datasets resulting in resource and compute efficiency for downstream processing constitutes an important area of study. "Intelligent compression" or deduplication in streaming scenarios,for precise identification and elimination of duplicates from the unbounded datastream is a greater challenge given the realtime nature of data arrival.Stable Bloom Filters(SBF) address this problem to a certain extent.However,SBF suffers from a high false negative rate(FNR) and slow convergence rate,thereby rendering it inefficient for applications with low FNR tolerance.In this paper, we present a novel Reservoir Sampling based Bloom Filter,(RSBF) data structure,based on the combined concepts of reservoir sampling and Bloom filters for approximate detection of duplicates in data streams.Using detailed theoretical analysis we prove analytical bounds on its false positive rate(FPR),false negative rate(FNR) and convergence rates with low memory requirements.We show that RSBF offers the currently lowest FN and convergence rates,and are better than those of SBF while using the same memory.Using empirical analysis on real-world datasets(3 million records) and synthetic datasets with around 1 billion records,we demonstrate upto 2x improvement in FNR with better convergence rates as compared to SBF,while exhibiting comparable FPR.To the best of our knowledge,this is the first attempt to integrate reservoir sampling method with Bloom filters for deduplication in streaming scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
