基于统计假设的快速爬行学习机制分析
Analysis of a Statistical Hypothesis Based Learning Mechanism for Faster crawling
摘要：
万维网（WWW）的发展使其从一个无形的网页扩展到一个巨大的网络信息中心，这逐渐增加了搜索引擎中抓取过程的复杂性。搜索引擎处理来自世界各地的大量查询，其答案完全取决于它通过爬行收集的知识。信息共享已成为社会的一种普遍现象，其主要方式是在网络上发布结构化、半结构化和非结构化的信息资源。这种社会实践导致网络资源呈指数级增长，因此在任何情况下，爬行以持续更新网络知识和修改现有资源变得至关重要。本文提出了一种基于统计假设的学习机制，用于学习不同网络环境下的爬行速度行为，并对爬行速度进行智能控制。缩放技术是用来比较性能提出的方法与标准的爬虫。实验结果表明，该算法具有很高的检索速度，并分析了在这种高速下对相关Web资源的检索。
英文摘要：
The growth of world-wide-web (WWW) spreads its wings from an intangible quantities of web-pages to a gigantic hub of web information which gradually increases the complexity of crawling process in a search engine. A search engine handles a lot of queries from various parts of this world, and the answers of it solely depend on the knowledge that it gathers by means of crawling. The information sharing becomes a most common habit of the society, and it is done by means of publishing structured, semi-structured and unstructured resources on the web. This social practice leads to an exponential growth of web-resource, and hence it became essential to crawl for continuous updating of web-knowledge and modification of several existing resources in any situation. In this paper one statistical hypothesis based learning mechanism is incorporated for learning the behavior of crawling speed in different environment of network, and for intelligently control of the speed of crawler. The scaling technique is used to compare the performance proposed method with the standard crawler. The high speed performance is observed after scaling, and the retrieval of relevant web-resource in such a high speed is analyzed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
