按需大数据集成：可重复科学研究的混合ETL方法
On-Demand Big Data Integration: A Hybrid ETL Approach for Reproducible Scientific Research
摘要：
科学研究需要访问、分析和共享分布在互联网规模的各种异构数据源中的数据。一个急切的ETL过程的第一步是构建一个集成的数据存储库，从数据源中集成和加载整个数据。这个过程的自举对于需要从非常大且通常众多的分布式数据源访问数据的科学研究来说是低效的。惰性ETL进程只加载元数据，但仍然是急切地加载。懒惰ETL在引导中更快。但是，由于预先提供了整个数据，因此在eager ETL的集成数据存储库上的查询执行得更快。在本文中，我们提出了一种新的ETL方法的科学数据集成，作为一个混合的渴望和懒惰的ETL方法，并适用于数据以及元数据。这样，混合ETL支持增量集成和加载来自数据源的元数据和数据。我们采用了一个人在回路的方法，以提高混合ETL，有选择性的数据集成驱动的用户查询和用户之间的集成数据共享。我们实现了我们的混合ETL方法在一个原型平台，Obidos，并评估它的背景下，医学研究的数据共享。Obidos通过选择性加载数据和元数据，同时将集成的数据存储在可扩展的集成数据存储库中，在科学研究数据集成和共享方面优于eager ETL和lazy ETL方法。
英文摘要：
Scientific research requires access, analysis, and sharing of data that is distributed across various heterogeneous data sources at the scale of the Internet. An eager ETL process constructs an integrated data repository as its first step, integrating and loading data in its entirety from the data sources. The bootstrapping of this process is not efficient for scientific research that requires access to data from very large and typically numerous distributed data sources. a lazy ETL process loads only the metadata, but still eagerly. Lazy ETL is faster in bootstrapping. However, queries on the integrated data repository of eager ETL perform faster, due to the availability of the entire data beforehand. In this paper, we propose a novel ETL approach for scientific data integration, as a hybrid of eager and lazy ETL approaches, and applied both to data as well as metadata. This way, Hybrid ETL supports incremental integration and loading of metadata and data from the data sources. We incorporate a human-in-the-loop approach, to enhance the hybrid ETL, with selective data integration driven by the user queries and sharing of integrated data between users. We implement our hybrid ETL approach in a prototype platform, Obidos, and evaluate it in the context of data sharing for medical research. Obidos outperforms both the eager ETL and lazy ETL approaches, for scientific research data integration and sharing, through its selective loading of data and metadata, while storing the integrated data in a scalable integrated data repository.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
