自动验证：使用从数据湖推断的数据域模式进行无监督数据验证
Auto-Validate: Unsupervised Data Validation Using Data-Domain Patterns Inferred from Data Lakes
摘要：
复杂的数据管道在BI报告和ML建模等各种应用中越来越常见。这些流水线经常定期重现（例如，每天或每周），因为BI报告需要刷新，ML模型需要重新训练。然而，据广泛报道，在复杂的生产管道中，上游数据馈送可能会以意想不到的方式发生变化，导致下游应用程序静默中断，解决成本高昂。因此，数据验证已成为一个重要的话题，谷歌和亚马逊最近的努力证明了这一点，其目标是在管道中出现数据质量问题时尽早发现这些问题。然而，我们在生产数据上的经验表明，在字符串值数据上，这些现有的方法会产生很高的假阳性率，并且经常需要人为干预。在这项工作中，我们开发了一种语料库驱动的方法来自动验证\emph{machine-generated data}，通过推断合适的数据验证“模式”，准确地描述了底层数据域，最大限度地减少误报，同时最大限度地提高数据质量问题。使用来自真实数据湖的生产数据进行的评估表明，Auto-Doppler比现有方法有效得多。该技术的一部分作为Microsoft Azure Purview中的自动标记功能提供。
英文摘要：
Complex data pipelines are increasingly common in diverse applications such as BI reporting and ML modeling. These pipelines often recur regularly (e.g., daily or weekly), as BI reports need to be refreshed, and ML models need to be retrained. However, it is widely reported that in complex production pipelines, upstream data feeds can change in unexpected ways, causing downstream applications to break silently that are expensive to resolve. Data validation has thus become an important topic, as evidenced by notable recent efforts from Google and Amazon, where the objective is to catch data quality issues early as they arise in the pipelines. Our experience on production data suggests, however, that on string-valued data, these existing approaches yield high false-positive rates and frequently require human intervention. In this work, we develop a corpus-driven approach to auto-validate \emph{machine-generated data} by inferring suitable data-validation "patterns" that accurately describe the underlying data domain, which minimizes false positives while maximizing data quality issues caught. Evaluations using production data from real data lakes suggest that Auto-Validate is substantially more effective than existing methods. Part of this technology ships as an Auto-Tag feature in Microsoft Azure Purview.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
