企业数据科学平台：一种联合数据访问的统一架构
Enterprise Data Science Platform: A Unified Architecture for Federated Data Access
AI总结：
EDSP通过统一架构实现多查询引擎环境下的联合数据访问，减少数据重复和供应商锁定问题，提升数据共享效率。
摘要：
企业机构在跨采用不同数据分析平台的部门之间共享数据时面临困难。如果n个数据集必须服务于m个环境，最多可以出现n*m个副本，这会增加不一致性和成本。传统仓库将数据复制到供应商特定的存储中；跨平台访问很困难。本研究提出企业数据科学平台（EDSP），其基于数据湖仓架构，并遵循“写一次，读任何地方”的原则。EDSP使多查询引擎环境能够实现联合数据访问，旨在针对具有周期性数据更新和查询响应时间从秒到分钟的数据科学工作负载。通过提供集中式数据管理，使多个查询引擎能够从相同的数据源进行联合访问，EDSP消除了传统数据仓库中固有的数据重复和供应商锁定问题。该平台采用四层架构：数据准备、数据存储、访问接口和查询引擎。这种设计强制分离关注点，并减少了在整合额外分析环境时的数据迁移需求。实验结果表明，主要的云数据仓库和编程环境可以直接查询EDSP管理的数据集。我们实现了并部署了EDSP到生产环境，确认了跨多个查询引擎的互操作性。对于跨不同分析环境的数据共享，EDSP相比传统方法需要数据迁移的常规方法，实现了33-44%的运营步骤减少。虽然查询延迟可能比原生表增加高达2.6倍，但端到端完成时间仍保持在秒级，维持了分析用例的实用性。基于我们的生产经验，EDSP为解决多查询引擎环境中的数据孤岛问题提供了实用的设计指南。
英文摘要：
Organizations struggle to share data across departments that have adopted different data analytics platforms. If n datasets must serve m environments, up to n*m replicas can emerge, increasing inconsistency and cost. Traditional warehouses copy data into vendor-specific stores; cross-platform access is hard. This study proposes the Enterprise Data Science Platform (EDSP), which builds on data lakehouse architecture and follows a Write-Once, Read-Anywhere principle. EDSP enables federated data access for multi-query engine environments, targeting data science workloads with periodic data updates and query response times ranging from seconds to minutes. By providing centralized data management with federated access from multiple query engines to the same data sources, EDSP eliminates data duplication and vendor lock-in inherent in traditional data warehouses. The platform employs a four-layer architecture: Data Preparation, Data Store, Access Interface, and Query Engines. This design enforces separation of concerns and reduces the need for data migration when integrating additional analytical environments. Experimental results demonstrate that major cloud data warehouses and programming environments can directly query EDSP-managed datasets. We implemented and deployed EDSP in production, confirming interoperability across multiple query engines. For data sharing across different analytical environments, EDSP achieves a 33-44% reduction in operational steps compared with conventional approaches requiring data migration. Although query latency may increase by up to a factor of 2.6 compared with native tables, end-to-end completion times remain on the order of seconds, maintaining practical performance for analytical use cases. Based on our production experience, EDSP provides practical design guidelines for addressing the data-silo problem in multi-query engine environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
