发表机构
Saint-Petersburg University（圣彼得堡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对匹配依赖发现算法HyMD的优化技术，采用新采样、泛化查找及依赖表示改进等方法，在Desbordante中实现优化版HyMD，实验证明其平均加速超40倍，改进版可双向集成Python供人使用。
AI 中文摘要
匹配依赖是函数依赖概念的推广，能应用自定义相似性函数匹配单个属性，在解决多种数据质量问题中有广泛应用，但发现过程计算量极大限制了实际应用。本文描述了针对HyMD（当前匹配依赖发现的最先进算法）的多种优化技术，包括新采样技术、更快的泛化查找技术和改进的依赖表示。通过在开源高性能数据探查器Desbordante中实现优化后的HyMD进行实验，结果表明平均比最先进实现加速超40倍，某些情况下超170倍。改进版HyMD可供任何人使用，具备双向Python集成。
英文摘要
Matching dependency is a generalization of the functional dependency concept, which allows users to apply custom similarity functions for matching individual attributes. Matching dependencies have a wide range of applications for solving various data quality problems, such as entity resolution, data deduplication, data integration, schema matching, and many more. However, their discovery is a very computationally intensive problem, which limits their practical application. In this paper, we describe a number of optimization techniques for HyMD - currently the state-of-the-art algorithm for the discovery of matching dependencies. These optimizations belong to both technical and scientific domains. The most important of them are: 1) a new sampling technique, 2) a faster generalization lookup technique, and 3) an improved representation of a dependency. The first one aims to raise the efficiency of inference from record pairs, while the last two are designed to speed up lattice-related operations. To evaluate our optimizations, we implemented our version of HyMD in Desbordante, an open-source high-performance data profiler. Experiments demonstrated that they allow for a speedup of more than 40x over the state-of-the-art implementation on average, reaching a speedup greater than 170x in some cases. Finally, the improved version of HyMD is ready to use by anyone. It comes with bidirectional Python integration, which allows calling the C++ algorithm implementation from Python programs while allowing users to supply their custom matching functions.
Journal ref2024 36th Conference of Open Innovations Association (FRUCT), Lappeenranta, Finland, 2024, pp. 729-740
DOI：10.23919/FRUCT64283.2024.10749955
