一种利用含噪声变化指示信号的可扩展爬取算法
A Scalable Crawling Algorithm Utilizing Noisy Change-Indicating Signals
AI总结：
针对网页更新爬取中利用含噪声变更辅助信号的挑战，提出一种可扩展的最优爬取算法，无需集中计算即可公平适配带宽、稳定速率，实验验证了其通用性。
摘要：
网页更新爬取是指在爬取器带宽有限的情况下，保持网页缓存的新鲜度，即当页面被请求时能提供最新版本的问题。在假设每个网页的变更事件和请求事件分别遵循独立泊松过程的前提下，Azar等人于2018年推导出了最优调度策略。本文研究该问题的扩展场景：可获得指示内容变更的辅助信息，例如各类网页ping信号，比如来自站点地图（sitemaps）、内容分发网络（CDN）等的信号。将这类辅助信息融入爬取策略存在挑战，原因在于：（1）信号可能存在噪声，包含误报事件且遗漏部分变更事件；（2）爬取器需在不同网页间实现公平的性能表现，而不同网页的辅助信息质量可能存在差异。我们提出了一种可扩展的爬取算法，该算法：（1）在温和假设下以最优方式利用含噪声的辅助信息；（2）无需大量集中式计算即可部署；（3）能够以恒定的总速率爬取网页，任意时间间隔内的总带宽使用都不会出现峰值，且当总带宽变化时无需集中式计算就能自动适配新的最优解。实验清晰地证明了我们方法的通用性。
英文摘要：
Web refresh crawling is the problem of keeping a cache of web pages fresh, that is, having the most recent copy available when a page is requested, given a limited bandwidth available to the crawler. Under the assumption that the change and request events, resp., to each web page follow independent Poisson processes, the optimal scheduling policy was derived by Azar et al. 2018. In this paper, we study an extension of this problem where side information indicating content changes, such as various types of web pings, for example, signals from sitemaps, content delivery networks, etc., is available. Incorporating such side information into the crawling policy is challenging, because (i) the signals can be noisy with false positive events and with missing change events; and (ii) the crawler should achieve a fair performance over web pages regardless of the quality of the side information, which might differ from web page to web page. We propose a scalable crawling algorithm which (i) uses the noisy side information in an optimal way under mild assumptions; (ii) can be deployed without heavy centralized computation; (iii) is able to crawl web pages at a constant total rate without spikes in the total bandwidth usage over any time interval, and automatically adapt to the new optimal solution when the total bandwidth changes without centralized computation. Experiments clearly demonstrate the versatility of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
