arXiv 1706.047222017-06cs.CYcs.DCcs.DS

一种用于处理海量传输源的流分析工作流的设计

The design of a streaming analytical workflow for processing massive transit feeds

Hung Cao, Monica Wachowicz

备注：: data ingestion, data contextualization, Hadoop MapReduce, transit networks, streaming data analytics, mobility context. Present at the 2nd International Symposium on Spatiotemporal Computing August 7-9 2017 at Harvard University, Cambridge, Massachusetts

摘要：

检索和分析交通馈送依赖于使用分析工作流，这些工作流可以处理与理解交通网络动态相关的大量数据流，这些动态在其发生的地理空间中完全是确定性的。在本文中，我们考虑的基本问题，在开发一个流的分析工作流程，分析连续到达的多个，无限的过境数据饲料自动处理和丰富他们的额外信息，包含更高层次的概念，相应于一个特定的流动性上下文。该工作流由三个任务组成：（1）用于创建时间窗口的流数据检索;（2）用于处理丢失数据、重叠数据或冗余数据的数据清理;以及（3）用于计算实际到达和离开时间以及巴士旅行期间的停止和移动的数据上下文化，并且还执行移动性上下文计算。该工作流在Hadoop云生态系统中使用来自蒙克顿市CODIAC交通系统的数据流实现。MapReduce的Map（）函数用于检索数据流并将其捆绑到多个集群中，这些集群随后由Reduce（）函数以并行方式处理，以便执行数据上下文化步骤。结果验证了云计算实现高性能和可扩展性的必要性，但是，由于计算和网络的延迟，很明显，数据清理任务不应仅使用云环境进行部署，为在不久的将来将其与雾计算相结合铺平了道路。

英文摘要：

Retrieving and analyzing transit feeds relies on working with analytical workflows that can handle the massive volume of data streams that are relevant to understand the dynamics of transit networks which are entirely deterministic in the geographical space in which they takes place. In this paper, we consider the fundamental issues in developing a streaming analytical workflow for analyzing the continuous arrival of multiple, unbounded transit data feeds for automatically processing and enriching them with additional information containing higher level concepts accordingly to a particular mobility context. This workflow consists of three tasks: (1) stream data retrieval for creating time windows; (2) data cleaning for handling missing data, overlap data or redundant data; and (3) data contextualization for computing actual arrival and departure times as well as the stops and moves during a bus trip, and also performing mobility context computation. The workflow was implemented in a Hadoop cloud ecosystem using data streams from the CODIAC Transit System of the city of Moncton, NB. The Map() function of MapReduce is used to retrieve and bundle data streams into numerous clusters which are subsequently handled in a parallel manner by the Reduce() function in order to execute the data contextualization step. The results validate the need for cloud computing for achieving high performance and scalability, however, due to the delay in computing and networking, it is clear that data cleaning tasks should not only be deployed using a cloud environment, paving the way to combine it with fog computing in the near future.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。