arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1208.4176cs.DB
收藏

为流处理应用建立用户自定义的自适应机制

Building User-defined Runtime Adaptation Routines for Stream Processing Applications

Gabriela Jacques-Silva, Buğra Gedik, Rohit Wagle, Kun-Lung Wu, Vibhore Kumar

摘要:

流处理应用程序被部署为连续查询,从提交到取消都在运行。这种部署模式限制了需要应用程序执行运行时调整的开发人员,例如算法调整、增量作业部署和特定于应用程序的故障恢复。目前,开发人员通过使用外部脚本和/或通过将与数据处理逻辑无关的运算符插入流处理图中来进行运行时适配。在本文中,我们描述了一个名为orchestrator的组件,它允许用户编写自动适应应用程序的运行时条件的例程。开发人员通过注册和处理事件以及指定驱动来构建编排器。事件可以由于系统状态的改变而生成(例如,应用组件故障),内置系统度量(例如,连接的吞吐量),或定制应用度量(例如,质量评分)。一旦协调器接收到事件,用户就可以通过使用协调器驱动API来采取适配动作。我们演示了使用的编排在IBM的System S中的三个不同的应用程序的上下文中,说明应用程序适应传入的数据分布的变化,应用程序故障,并按需动态组合。

英文摘要:

Stream processing applications are deployed as continuous queries that run from the time of their submission until their cancellation. This deployment mode limits developers who need their applications to perform runtime adaptation, such as algorithmic adjustments, incremental job deployment, and application-specific failure recovery. Currently, developers do runtime adaptation by using external scripts and/or by inserting operators into the stream processing graph that are unrelated to the data processing logic. In this paper, we describe a component called orchestrator that allows users to write routines for automatically adapting the application to runtime conditions. Developers build an orchestrator by registering and handling events as well as specifying actuations. Events can be generated due to changes in the system state (e.g., application component failures), built-in system metrics (e.g., throughput of a connection), or custom application metrics (e.g., quality score). Once the orchestrator receives an event, users can take adaptation actions by using the orchestrator actuation APIs. We demonstrate the use of the orchestrator in IBM's System S in the context of three different applications, illustrating application adaptation to changes on the incoming data distribution, to application failures, and on-demand dynamic composition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑