学习爬行
Learning to Crawl
摘要:
Web爬行是保持网页缓存新鲜的问题,即,当一个页面被请求时,有最新的可用副本。这个问题通常与网络爬虫可用的带宽有限的自然限制相结合。Azar等人[2018]最佳地解决了相应的优化问题,假设对于每个网页,两次更改之间的经过时间和两次请求之间的经过时间都遵循具有已知参数的泊松分布。在本文中,我们研究相同的控制问题,但假设变化率是先验未知的,因此我们需要仅使用部分观测值(即,指示页面自上次刷新以来是否已经改变的单比特信号)。作为一个出发点,我们讨论的条件下,人们可以解决这个问题的部分可观性。接下来,我们提出了一个实用的估计和计算置信区间的观察之间的流逝时间。最后,我们表明,探索和提交算法实现了$\mathcal{O}(\sqrt{T})$遗憾,仔细选择的探索视野。我们的模拟研究表明,我们的在线政策的规模以及实现接近最优性能的参数范围很广。
英文摘要:
Web crawling is the problem of keeping a cache of webpages fresh, i.e., having the most recent copy available when a page is requested. This problem is usually coupled with the natural restriction that the bandwidth available to the web crawler is limited. The corresponding optimization problem was solved optimally by Azar et al. [2018] under the assumption that, for each webpage, both the elapsed time between two changes and the elapsed time between two requests follow a Poisson distribution with known parameters. In this paper, we study the same control problem but under the assumption that the change rates are unknown a priori, and thus we need to estimate them in an online fashion using only partial observations (i.e., single-bit signals indicating whether the page has changed since the last refresh). As a point of departure, we characterise the conditions under which one can solve the problem with such partial observability. Next, we propose a practical estimator and compute confidence intervals for it in terms of the elapsed time between the observations. Finally, we show that the explore-and-commit algorithm achieves an $\mathcal{O}(\sqrt{T})$ regret with a carefully chosen exploration horizon. Our simulation study shows that our online policy scales well and achieves close to optimal performance for a wide range of the parameters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。