arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.06356cs.CR
收藏

CRATOR:一个暗网爬虫

CRATOR: a Dark Web Crawler

Tilburg University(蒂尔堡大学);TU/e(埃因霍温理工大学)

Jessica De Pascale, Giuseppe Cascavilla, Damian A. Tamburri, Willem-Jan Van Den Heuvel

AI总结:

研究提出暗网爬虫CRATOR,结合种子URL列表、链接分析等方法,能高效提取处理安全协议的页面,还通过用户代理轮换等保持匿名性,经评估其可有效用于多种网络相关应用。

摘要:

暗网爬取是一个复杂的过程,涉及特定方法和技术来浏览Tor网络并从隐藏服务中提取数据。本研究提出了一个通用暗网爬虫,旨在高效提取处理安全协议(如验证码)的页面。我们的方法结合种子URL列表、链接分析和扫描来发现新内容,还纳入了用户代理轮换和代理使用方法以保持匿名性并避免被检测。我们用覆盖率、性能和鲁棒性等指标评估爬虫有效性。结果表明,我们的爬虫能有效提取处理安全协议的页面,同时保持匿名性并避免被检测,可用于威胁情报、网络安全和在线调查等各种应用。

英文摘要:

Dark web crawling is a complex process that involves specific methodologies and techniques to navigate the Tor network and extract data from hidden services. This study proposes a general dark web crawler designed to extract pages handling security protocols, such as captchas, efficiently. Our approach uses a combination of seed URL lists, link analysis, and scanning to discover new content. We also incorporate methods for user-agent rotation and proxy usage to maintain anonymity and avoid detection. We evaluate the effectiveness of our crawler using metrics such as coverage, performance and robustness. Our results demonstrate that our crawler effectively extracts pages handling security protocols while maintaining anonymity and avoiding detection. Our proposed dark web crawler can be used for various applications, including threat intelligence, cybersecurity, and online investigations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑