在内存有限的磁盘驻留数据上高效发现唯一列组合

中文摘要 AI 辅助

唯一列组合（UCC）发现是数据概览的核心任务，用于描述表的键约束。现有算法因高内存消耗和计算成本，难以高效处理大规模磁盘驻留数据。本文受UCC发现与横截超图关系的启发，提出一种新颖的DUD算法，用于在内存有限的磁盘驻留数据上高效发现UCC。该算法不采用二次复杂度的完整差集生成，仅生成部分差集用于超图构建，随后通过最小击中集枚举生成候选集并进行验证；其设计了一种策略，通过对某些选定属性取值相同的元组进行成对比较，生成全部有用差集；本文还提出并证明了一个新定理，可直接将包含选定属性的候选集报告为真实UCC而无需验证，大幅减少需验证的候选数量；此外，设计了一种基于哈希的批量验证策略，用于在关系实例上验证候选集，每次仅需在内存中维护少量元组。在合成数据集和真实数据集上开展的大量实验结果表明，DUD可高效且低内存地在磁盘驻留数据上发现UCC。

英文摘要

The discovery of unique column combinations (UCCs) is a core task in data profiling, describing the key constraints of a table. The existing algorithms cannot deal with large-scale disk-resident data well due to high memory consumption and computational cost. In this paper, a novel DUD algorithm is developed to efficiently discover UCCs on disk-resident data with limited memory, which is inspired by the relationship between UCC discovery and transversal hypergraph. Rather than complete difference set generation of quadratic complexity, DUD only generates partial difference sets for hypergraph construction, followed by minimal hitting set enumeration to generate candidates and a validation process. DUD devises a strategy to generate full useful difference sets by pairwise comparisons of tuples having the same values with respect to some selected attributes. A novel theorem is developed and proved in this paper to report the candidates including the selected attributes as true UCCs directly without validation, which reduces the number of candidates to be validated significantly. A hash-based batch validation strategy is devised to validate a set of candidates on the relation instance, which only needs to maintain a small number of tuples in memory at a time. The extensive experimental results, conducted on synthetic and real-life data sets, show that DUD can discover UCCs on disk-resident data with high efficiency and low memory consumption.

发表机构

• School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）

机构由 AI 辅助整理，请以论文原文为准。