arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.20077cs.LGcs.DCcs.SE
收藏

托管地理分布式特征存储:架构与系统设计

Managed Geo-Distributed Feature Store: Architecture and System Design

Microsoft(微软)

Anya Li, Bhala Ranganathan, Feng Pan, Mickey Zhang, Qianjun Xu, Runhan Li, Sethu Raman, Shail Paragbhai Shah, Vivienne Tang

AI总结:

本文针对企业特征工程中特征冲突、重复、搜索重用困难及线上线下偏斜等问题,提出托管地理分布式特征存储的核心架构组件与系统设计经验。

摘要:

企业正在使用机器学习解决现实问题,并在此过程中开发数百到数千个特征。它们正在构建特征工程流水线,作为 MLOps 生命周期的一部分,以转换来自各种数据源的数据,并将其物化以供未来使用。如果没有特征存储,不同业务组中的不同团队将独立维护上述过程,这可能导致系统中出现冲突和重复的特征。数据科学家发现很难搜索和重用现有特征,并且维护版本控制也很痛苦。此外,与在线(推理)-离线(训练)偏斜和数据泄漏相关的特征正确性违规很常见。尽管机器学习社区已经广泛讨论了特征存储的必要性及其目的,但本文旨在捕捉构成托管特征存储的核心架构组件,并分享构建此类系统的设计经验。

英文摘要:

Companies are using machine learning to solve real-world problems and are developing hundreds to thousands of features in the process. They are building feature engineering pipelines as part of MLOps life cycle to transform data from various data sources and materialize the same for future consumption. Without feature stores, different teams across various business groups would maintain the above process independently, which can lead to conflicting and duplicated features in the system. Data scientists find it hard to search for and reuse existing features and it is painful to maintain version control. Furthermore, feature correctness violations related to online (inferencing) - offline (training) skews and data leakage are common. Although the machine learning community has extensively discussed the need for feature stores and their purpose, this paper aims to capture the core architectural components that make up a managed feature store and to share the design learning in building such a system.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑