arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1205.2891cs.IR
收藏

利用Web爬行实现Web上信息检索的有效性

Effective performance of information retrieval on web by using web crawling

Sk. AbdulNabi, P. Premchand

摘要:

万维网由超过500亿个在线页面组成。它是高度动态的,即网络不断引入新的功能,吸引了许多人。由于这种规模上的爆炸,有效的信息检索系统或搜索引擎可用于访问信息。在本文中,我们提出了EPOW(有效性能的网络爬虫)架构。它是一个软件代理,其主要目标是最大限度地减少用户定位所需信息的过载。我们设计了网络爬虫,考虑到并行化的政策。由于我们的EPOW爬虫有一个高度优化的系统,它可以每秒下载大量的页面,同时对崩溃鲁棒。我们还提出了使用数据结构的概念来实现调度器和循环队列,以提高我们的网络爬虫的性能。

英文摘要:

World Wide Web consists of more than 50 billion pages online. It is highly dynamic i.e. the web continuously introduces new capabilities and attracts many people. Due to this explosion in size, the effective information retrieval system or search engine can be used to access the information. In this paper we have proposed the EPOW (Effective Performance of WebCrawler) architecture. It is a software agent whose main objective is to minimize the overload of a user locating needed information. We have designed the web crawler by considering the parallelization policy. Since our EPOW crawler has a highly optimized system it can download a large number of pages per second while being robust against crashes. We have also proposed to use the data structure concepts for implementation of scheduler & circular Queue to improve the performance of our web crawler.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑