arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.07335cs.DC
收藏

顶- $K$负载下最优冷热层配置的秘书雇用问题

Adapting The Secretary Hiring Problem for Optimal Hot-Cold Tier Placement under Top-$K$ Workloads

Ben Blamey, Fredrik Wrede, Johan Karlsson, Andreas Hellander, Salman Toor

摘要:

Top-K查询是信息检索中的一种启发式查询。本文提出了一种方法,最佳分层存储分配流处理工作负载下使用这种启发式:那些只需要分析顶部的$K$排名最相关的,或最有趣的,文件从一个固定长度的流,流窗口,或批处理作业。在该工作流中,利用用户指定的兴趣度函数来分析文档的相关性,基于该兴趣度函数对文档进行排名,选择(并因此存储)顶部$K$以用于进一步处理。此工作流程允许人在回路系统中,包括监督机器学习,以优先考虑文档。这种情况与经典的秘书招聘问题(SHP)相似,并且文档写入的预期速率和文档生命周期可以建模为文档索引的函数。我们提出了基于参数的算法存储层的位置,最大限度地减少文件的存储和运输成本。我们表明,最佳参数值是这些成本的函数。可以为这类工作负载先验地建模应用程序IO行为。当与分层存储结合使用时,IO概率模型的易处理性使得优化层分配变得容易,从而产生简单的分析解决方案。此相反分层存储优化方面的现有工作(通常很复杂),这些工作要么与特定使用情形紧密结合,要么需要主动监视应用程序IO负载(反应式方法)-不适合科学计算领域中常见的长期运行或一次性操作。我们用生物化学模型探索的跟踪驱动模拟来验证我们的模型,并给出了两个云存储案例研究的工作示例。

英文摘要:

Top-K queries are an established heuristic in information retrieval. This paper presents an approach for optimal tiered storage allocation under stream processing workloads using this heuristic: those requiring the analysis of only the top-$K$ ranked most relevant, or most interesting, documents from a fixed-length stream, stream window, or batch job. In this workflow, documents are analyzed relevance with a user-specified interestingness function, on which they are ranked, the top-$K$ being selected (and hence stored) for further processing. This workflow allows human in the loop systems, including supervised machine learning, to prioritize documents. This scenario bears similarity to the classic Secretary Hiring Problem (SHP), and the expected rate of document writes, and document lifetime, can be modelled as a function of document index. We present parameter-based algorithms for storage tier placement, minimizing document storage and transport costs. We show that optimal parameter values are a function of these costs. It is possible to model application IO behaviour a priori for this class of workloads. When combined with tiered storage, the tractability of the probabilistic model of IO makes it easy to optimize tier allocation, yielding simple analytic solutions. This contrasts with (often complex) existing work on tiered storage optimization, which is either tightly coupled to specific use cases, or requires active monitoring of application IO load (a reactive approach) -- ill-suited to long-running or one-off operations common in the scientific computing domain.We validate our model with a trace-driven simulation of a bio-chemical model exploration, and give worked examples for two cloud storage case studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑