发表机构
Hangzhou Dianzi University; China Mobile (Suzhou) Software Technology Co., Ltd.; Nine Chapters (Zhejiang) Technology Co., Ltd.（杭州电子科技大学; 中国移动（苏州）软件技术有限公司; 九章（浙江）科技有限公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出一种基于预剥离计数的自调整规模IBLT集合协调协议，可在解码前测量差基数，通过两次交互轮完成协调，经实测验证了其有效性。
AI 中文摘要
集合协调以远低于数据量的通信量恢复对称差A△B，可逆布鲁姆查找表（IBLT）是标准工具，但其容量必须匹配未知的差基数d。在数据库部署中，d跨度达数个数量级，因此容量不足会导致重新扫描或重放，而容量过大则会浪费网络和内存资源。我们证明IBLT在解码前即可测量d：每个单元计数记录映射到该单元的差元素数量，计数数组上的二次统计量可提供无额外传输字节的精确无偏估计，该估计因从同一草图读取，故在解码失败后仍有效。对于标准IBLT，我们推导了精确的均值和方差公式及卡方置信区间，相同的卡方定律适用于深度过载下的失败实例。映射感知定理通过每个变体适配器将该构造扩展至不规则、无速率和MET IBLT。我们的自调整规模协议利用第一轮失败一步计算第二轮容量，在两次交互轮内完成，并有明确的成功概率保证。我们在九数云对90天内的41603次协调运行进行了分析，在生产表跨引擎重放中与默克尔风格定位进行了端到端比较，并在中国移动的Redis和Pika间部署。因此，第一轮将未知差基数转化为已付出成本的测量。
英文摘要
Cross-system data replication pipelines cannot confirm end-to-end consistency from the local guarantees of each hop, so the two endpoints must be compared directly on a periodic basis. Once the rows of a fixed snapshot are normalized into fingerprints, the task reduces to finding the symmetric difference of the two sets. An Invertible Bloom Lookup Table (IBLT) reconciles the sets with communication that grows only with the difference cardinality $d$, independent of table size, but its capacity must be fixed while $d$ is still unknown. Across 41,603 production reconciliations over 90 days, nonzero $d$ spans about seven orders of magnitude, and no reliable empirical constant exists. We show that the count array of an IBLT has already measured $d$ before decoding. The measurement is in-band: it is carried by the recovery sketch itself and adds no bytes dedicated to estimation. A mapping-aware theorem extends the construction to Irregular, Rateless, and MET IBLTs. The protocol reads the estimate only after a decoding failure; we prove that the failure-conditioned lower quantile bounds the risk of underestimation, which gives the second-round capacity a configurable success-probability guarantee. The resulting self-sizing protocol attempts recovery with a small first-round sketch and stops on success; on failure it reads $d$, sizes the second round, and completes reconciliation in at most two rounds. Against a controlled oracle, communication is 1.29-1.47 times that of a scheme given $d$ in advance. Production workload characterization, relational-database replay, and a cross-city KV deployment confirm the end-to-end mechanism. In production on an Oracle-MySQL link, all completed runs succeeded within two rounds, over 90% on the 1-RTT fast path with a single 16 KB sketch.
CommentsCode and archived artifact: https://github.com/whitewum/self-sizing
