使用回归技术预测大数据传输
Using Regression Techniques to Predict Large Data Transfers
摘要：
最近数据网格的激增和使用资源作为分布式数据存储的日益普遍的做法为研究人员社区提供了一个方便的环境，以共享，复制和管理对大型数据集副本的访问。这就产生了一个问题，即哪一个副本的访问效率最高。在这种环境中，从多个副本位置之一获取数据需要准确预测端到端传输时间。这个问题的答案可能取决于许多因素，包括资源的物理特性以及CPU、网络和存储设备上的负载行为，这些设备是连接可能的源和接收器的端到端数据路径的一部分。我们的方法结合了端到端的应用程序吞吐量的观察与网络和磁盘负载的变化，并捕捉整个系统的性能和负载模式的变化。我们的预测特征的几个共享设备（网络和磁盘）的负载变化对文件传输时间的影响。我们开发了一套单变量和多变量预测器，可以使用多个数据源来提高预测的准确性，并解决数据网格的变化（数据的可用性和传输的偶发性）。我们使用GridFTP运行了大量的数据传输实验，并观察到我们的测试平台站点的性能预测误差在15%以内，这对于实用的系统来说是非常有希望的。
英文摘要：
The recent proliferation of Data Grids and the increasingly common practice of using resources as distributed data stores provide a convenient environment for communities of researchers to share, replicate, and manage access to copies of large datasets. This has led to the question of which replica can be accessed most efficiently. In such environments, fetching data from one of the several replica locations requires accurate predictions of end-to-end transfer times. The answer to this question can depend on many factors, including physical characteristics of the resources and the load behavior on the CPUs, networks, and storage devices that are part of the end-to-end data path linking possible sources and sinks. Our approach combines end-to-end application throughput observations with network and disk load variations and captures whole-system performance and variations in load patterns. Our predictions characterize the effect of load variations of several shared devices (network and disk) on file transfer times. We develop a suite of univariate and multivariate predictors that can use multiple data sources to improve the accuracy of the predictions as well as address Data Grid variations (availability of data and sporadic nature of transfers). We ran a large set of data transfer experiments using GridFTP and observed performance predictions within 15% error for our testbed sites, which is quite promising for a pragmatic system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
