具有延迟反馈和衰减代价的MAB缓存替换
Cache Replacement as a MAB with Delayed Feedback and Decaying Costs
摘要：
受缓存替换问题的启发，我们提出并解决了一个新的变种的著名的多臂强盗（MAB），从而提供了一个解决方案，以改善现有的国家的最先进的缓存管理方法。每个arm（或expert）代表一个不同的缓存替换策略，该策略建议页面在需要时从缓存中驱逐。关于驱逐的反馈以“未命中”的形式出现，但在采取行动后的不确定时间，并且驱逐的成本被设置为与响应时间成正比。如果反馈在延迟的阈值之后出现，则会被忽略，我们将延迟设置为等于页面回收历史的大小。因此，对于超过阈值的延迟，其成本被假定为零。因此，我们称之为延迟反馈和衰减成本问题。我们引入了一个自适应强化学习算法EXP 4-DFDC，提供了一个解决问题的方案。我们推导出EXP 4-DFDC的最佳学习率，定义了探索和利用之间的平衡，并从理论上证明了我们的算法的预期遗憾是一个消失的量作为时间的函数。作为一个应用程序，我们表明，LeCaR，最近表现最好的机器学习算法的缓存替换，可以增强自适应学习使用我们的配方。我们提出了一个改进的自适应版本的LeCaR，称为OLeCaR，与学习率集确定的理论推导，在这里提出，以尽量减少遗憾EXP 4-DFDC。因此，LeCaR和OLeCaR在理论上保证会随着时间的推移而消失。
英文摘要：
Inspired by the cache replacement problem, we propose and solve a new variant of the well-known multi-armed bandit (MAB), thus providing a solution for improving existing state-of-the-art cache management methods. Each arm (or expert) represents a distinct cache replacement policy, which advises on the page to evict from the cache when needed. Feedback on the eviction comes in the form of a "miss", but at an indeterminate time after the action is taken, and the cost of the eviction is set to be inversely proportional to the response time. The feedback is ignored if it comes after a threshold value for the delay, which we set to be equal to the size of the page eviction history. Thus, for delays beyond the threshold, its cost is assumed to be zero. Consequently, we call this problem with delayed feedback and decaying costs. We introduce an adaptive reinforcement learning algorithm EXP4-DFDC that provides a solution to the problem. We derive an optimal learning rate for EXP4-DFDC that defines the balance between exploration and exploitation and proves theoretically that the expected regret of our algorithm is a vanishing quantity as a function of time. As an application, we show that LeCaR, a recent top-performing machine learning algorithm for cache replacement, can be enhanced with adaptive learning using our formulations. We present an improved adaptive version of LeCaR, called OLeCaR, with the learning rate set as determined by the theoretical derivation presented here to minimize regret for EXP4-DFDC. It then follows that LeCaR and OLeCaR are theoretically guaranteed to have vanishing regret over time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
