面向机器学习的数据集管理平台
Dataset Management Platform for Machine Learning
摘要:
数据集中的数据质量可以对使用数据集训练和/或评估的机器学习模型的性能具有实质性影响。有效的数据集管理,包括数据清理、版本控制、访问控制、数据集转换、自动化、完整性和安全性等任务,可以帮助提高机器学习过程的效率和速度。目前,工程师花费大量的手动工作和时间来管理数据集版本或为机器学习任务准备数据集。本公开描述了一种有效地管理和使用数据集的平台。这些技术集成了数据集管理和数据集转换机制。描述了一个存储引擎,它充当所有数据的真实来源,并处理版本控制、访问控制等。数据集转换机制是生成数据集(快照)以满足不同目的的关键部分。所描述的技术可以支持不同的工作流、流水线或数据编排需求,例如,用于训练和/或评估机器学习模型。
英文摘要:
The quality of the data in a dataset can have a substantial impact on the performance of a machine learning model that is trained and/or evaluated using the dataset. Effective dataset management, including tasks such as data cleanup, versioning, access control, dataset transformation, automation, integrity and security, etc., can help improve the efficiency and speed of the machine learning process. Currently, engineers spend a substantial amount of manual effort and time to manage dataset versions or to prepare datasets for machine learning tasks. This disclosure describes a platform to manage and use datasets effectively. The techniques integrate dataset management and dataset transformation mechanisms. A storage engine is described that acts as a source of truth for all data and handles versioning, access control etc. The dataset transformation mechanism is a key part to generate a dataset (snapshot) to serve different purposes. The described techniques can support different workflows, pipelines, or data orchestration needs, e.g., for training and/or evaluation of machine learning models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。