文档陈旧的Web搜索引擎机器人最优阈值控制
Optimal Threshold Control by the Robots of Web Search Engines with Obsolescence of Documents
摘要：
一个典型的网络搜索引擎由三个主要部分组成：抓取引擎、索引引擎和搜索引擎。目前的工作旨在优化爬行引擎的性能。抓取引擎发现新的网页并更新存在于网络搜索引擎的数据库中的网页。爬行引擎有几个机器人从互联网上收集信息。我们首先计算系统的各种性能度量（例如，由于缓冲区溢出导致的任意页面丢失的概率、系统饥饿的概率、在缓冲区中等待的平均时间）。直观地说，我们希望避免系统饥饿，同时最大限度地减少信息损失。我们制定的问题作为一个多标准的优化问题，并赋予每个标准的权重。我们在阈值策略类中解决它。我们考虑了一个非常一般的网页到达过程的批标记马尔可夫到达过程和一个非常一般的服务时间模型的相位型分布。该模型已被应用于性能评估和优化的爬虫设计的INRIA Maestro团队在RIAM INRIA-Canon研究项目的框架。
英文摘要：
A typical web search engine consists of three principal parts: crawling engine, indexing engine, and searching engine. The present work aims to optimize the performance of the crawling engine. The crawling engine finds new web pages and updates web pages existing in the database of the web search engine. The crawling engine has several robots collecting information from the Internet. We first calculate various performance measures of the system (e.g., probability of arbitrary page loss due to the buffer overflow, probability of starvation of the system, the average time waiting in the buffer). Intuitively, we would like to avoid system starvation and at the same time to minimize the information loss. We formulate the problem as a multi-criteria optimization problem and attributing a weight to each criterion. We solve it in the class of threshold policies. We consider a very general web page arrival process modeled by Batch Marked Markov Arrival Process and a very general service time modeled by Phase-type distribution. The model has been applied to the performance evaluation and optimization of the crawler designed by INRIA Maestro team in the framework of the RIAM INRIA-Canon research project.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
