arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 1208.0798cs.DS
收藏

Biff(Bloom Filter)码:大数据集的快速纠错

Biff (Bloom Filter) Codes : Fast Error Correction for Large Data Sets

Michael Mitzenmacher, George Varghese

摘要:

大型数据集在云和虚拟化环境中越来越常见。例如,多个千兆字节的传输是常见的,这种大小的复制块也是如此。在这种情况下,即使预期的错误数量很小,也需要快速纠错或数据协调。出于这样的云协调问题,我们考虑为大数据设计的纠错方案,解释为什么以前的方法似乎不合适。我们介绍了Biff码,这是基于布隆过滤器,是专为大数据。对于具有长度为$L$和$E$错误的消息的Biff码,编码时间为$O(L)$,解码时间为$O(L + E)$,空间开销为$O(E)$。Biff码是低密度奇偶校验码;它们类似于Tornado码,但设计用于错误而不是擦除。此外,Biff码被设计得非常简单,去除了任何显式的图结构,并且完全基于哈希表。我们从最近开发的数据结构,可逆布隆查找表的一组和解算法,通过一个简单的减少Biff码。虽然基本理论非常简单,但使这种代码特别有吸引力的是它可以实现的容易性和解码的速度。我们提出了一个原型实现的结果,解码消息的100万个字,数千个错误,远远低于一秒。

英文摘要:

Large data sets are increasingly common in cloud and virtualized environments. For example, transfers of multiple gigabytes are commonplace, as are replicated blocks of such sizes. There is a need for fast error-correction or data reconciliation in such settings even when the expected number of errors is small. Motivated by such cloud reconciliation problems, we consider error-correction schemes designed for large data, after explaining why previous approaches appear unsuitable. We introduce Biff codes, which are based on Bloom filters and are designed for large data. For Biff codes with a message of length $L$ and $E$ errors, the encoding time is $O(L)$, decoding time is $O(L + E)$ and the space overhead is $O(E)$. Biff codes are low-density parity-check codes; they are similar to Tornado codes, but are designed for errors instead of erasures. Further, Biff codes are designed to be very simple, removing any explicit graph structures and based entirely on hash tables. We derive Biff codes by a simple reduction from a set reconciliation algorithm for a recently developed data structure, invertible Bloom lookup tables. While the underlying theory is extremely simple, what makes this code especially attractive is the ease with which it can be implemented and the speed of decoding. We present results from a prototype implementation that decodes messages of 1 million words with thousands of errors in well under a second.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑