为获取易逝内容设计的Whittle指数策略
Whittle Index Policy for Crawling Ephemeral Content
AI总结：
本文提出基于Whittle指数的高效爬虫策略，解决易逝内容调度问题，通过分解问题提升效率。
摘要：
本文考虑了调度爬虫获取多个网站上易逝内容的任务。用户通常在几天或几小时后对易逝内容失去兴趣，如社交媒体群组中的新闻或帖子。因此，为这类易逝信息源开发及时的爬取策略非常重要。我们首先将此问题 formulation 为一个具有平均奖励的最优控制问题。奖励可以以点击次数或相关搜索请求的数量来衡量。初始问题 formulation 遭受维度灾难的困扰，即使在中等数量的信息源情况下也迅速变得难以处理。幸运的是，该问题允许一个Whittle指数，这导致了问题的分解和一个非常简单且高效的爬取策略。我们推导了Whittle指数并提供了其理论依据。
英文摘要：
We consider a task of scheduling a crawler to retrieve content from several sites with ephemeral content. A user typically loses interest in ephemeral content, like news or posts at social network groups, after several days or hours. Thus, development of timely crawling policy for such ephemeral information sources is very important. We first formulate this problem as an optimal control problem with average reward. The reward can be measured in the number of clicks or relevant search requests. The problem in its initial formulation suffers from the curse of dimensionality and quickly becomes intractable even with moderate number of information sources. Fortunately, this problem admits a Whittle index, which leads to problem decomposition and to a very simple and efficient crawling policy. We derive the Whittle index and provide its theoretical justification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
