arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1202.1837cs.IRcs.SI
收藏

一种通过博客在线抓取实现连续Web监控的架构

A Proposed Architecture for Continuous Web Monitoring Through Online Crawling of Blogs

Mehdi Naghavi, Mohsen Sharifi

摘要:

及时了解网络空间中注册的内容,可以极大地帮助心理学家,营销人员和政治分析师熟悉,分析,做出决策并根据社会的不同需求正确行事。Web空间中的海量信息阻碍了我们对整个Web空间的持续在线调查。专注于所考虑的博客限制了我们的工作领域,并使Web空间中的在线爬行成为可能。本文提出了一种使用聚焦爬虫对相关博客进行连续在线爬虫的体系结构,并对所获得的数据进行调查和分析。在线抓取是根据ping服务器机器的最新公告完成的。该算法通过对关键词的定位形成一个加权图,使得聚焦爬虫可以根据加权图获取相关网页的完整文本。

英文摘要:

Getting informed of what is registered in the Web space on time, can greatly help the psychologists, marketers and political analysts to familiarize, analyse, make decision and act correctly based on the society`s different needs. The great volume of information in the Web space hinders us to continuously online investigate the whole space of the Web. Focusing on the considered blogs limits our working domain and makes the online crawling in the Web space possible. In this article, an architecture is offered which continuously online crawls the related blogs, using focused crawler, and investigates and analyses the obtained data. The online fetching is done based on the latest announcements of the ping server machines. A weighted graph is formed based on targeting the important key phrases, so that a focused crawler can do the fetching of the complete texts of the related Web pages, based on the weighted graph.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑