Vamsa：数据科学中的自动来源跟踪
Vamsa: Automated Provenance Tracking in Data Science Scripts
摘要：
由于各种机器学习应用程序的不言自明或监管要求，最近在机器学习（ML）模型的公平性，偏见和可解释性领域进行了大量研究。我们做了以下观察：所有这些方法都需要对ML模型和用于训练它们的数据之间的关系有一个强大的理解。在这项工作中，我们介绍了ML出处跟踪问题：基本思想是自动跟踪数据集中的哪些列已用于导出ML模型的特征/标签。我们讨论了在Python（数据科学家最常用的语言）的上下文中捕获这些信息的挑战。然后，我们介绍了Vamsa，一个模块化系统，它可以从Python脚本中提取出处，而不需要对用户的代码进行任何更改。使用26 K真实数据科学脚本，我们验证了Vamsa在覆盖率和性能方面的有效性。我们还评估Vamsa的准确性手动标记的数据的一个较小的子集。我们的分析表明，Vamsa的准确率和召回率范围从90.4%到99.1%，其延迟是在毫秒级的平均大小的脚本。根据我们在生产中部署ML模型的经验，我们还提供了一个示例，其中Vamsa帮助自动识别受数据损坏问题影响的模型。
英文摘要：
There has recently been a lot of ongoing research in the areas of fairness, bias and explainability of machine learning (ML) models due to the self-evident or regulatory requirements of various ML applications. We make the following observation: All of these approaches require a robust understanding of the relationship between ML models and the data used to train them. In this work, we introduce the ML provenance tracking problem: the fundamental idea is to automatically track which columns in a dataset have been used to derive the features/labels of an ML model. We discuss the challenges in capturing such information in the context of Python, the most common language used by data scientists. We then present Vamsa, a modular system that extracts provenance from Python scripts without requiring any changes to the users' code. Using 26K real data science scripts, we verify the effectiveness of Vamsa in terms of coverage, and performance. We also evaluate Vamsa's accuracy on a smaller subset of manually labeled data. Our analysis shows that Vamsa's precision and recall range from 90.4% to 99.1% and its latency is in the order of milliseconds for average size scripts. Drawing from our experience in deploying ML models in production, we also present an example in which Vamsa helps automatically identify models that are affected by data corruption issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
