这是一个本地域:从公共数据中积累国家代码顶级域
This Is a Local Domain: On Amassing Country-Code Top-Level Domains from Public Data
AI总结:
本研究利用证书透明度和Common Crawl等公共数据提取ccTLDs域名,发现其覆盖率达43%-80%,可作为网络普查中ccTLDs的可靠代理。
摘要:
域名列表是进行具有代表性的网络普查的关键要素。不幸的是,此类普查通常缺乏对国家级顶级域(ccTLDs)下域名的视角。这引入了不必要的偏差:许多国家拥有丰富的本地网络,如果其ccTLDs未被考虑,这些网络将保持隐藏。ccTLDs很少被考虑的原因是获取访问权限——如果可能的话——往往很费力。为了解决这个问题,我们提出疑问:我们能从公共来源中了解ccTLDs的什么信息?我们从6年的公共数据中提取ccTLDs下的域名,这些数据来自证书透明度日志和Common Crawl。我们将此与19个ccTLDs的基准真值进行比较,这些ccTLDs我们拥有完整的DNS区域。我们发现公共数据覆盖了这些ccTLDs的43%-80%,并且覆盖范围随时间增长。通过比较端口扫描数据,我们还表明这些公共来源揭示了ccTLDs下网络存在的显著部分。我们得出结论,在无法完全访问ccTLDs的情况下,从公共来源学习的域名在进行网络普查时可以作为一个良好的代理。
英文摘要:
Domain lists are a key ingredient for representative censuses of the Web. Unfortunately, such censuses typically lack a view on domains under country-code top-level domains (ccTLDs). This introduces unwanted bias: many countries have a rich local Web that remains hidden if their ccTLDs are not considered. The reason ccTLDs are rarely considered is that gaining access -- if possible at all -- is often laborious. To tackle this, we ask: what can we learn about ccTLDs from public sources? We extract domain names under ccTLDs from 6 years of public data from Certificate Transparency logs and Common Crawl. We compare this against ground truth for 19 ccTLDs for which we have the full DNS zone. We find that public data covers 43%-80% of these ccTLDs, and that coverage grows over time. By also comparing port scan data we then show that these public sources reveal a significant part of the Web presence under a ccTLD. We conclude that in the absence of full access to ccTLDs, domain names learned from public sources can be a good proxy when performing Web censuses.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。