衡量爬虫所见：纵向网络爬虫中的发现曲线、核心持久性和外壳动态
Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls
AI总结：
研究纵向网络爬虫，定义形式语言并通过发现曲线扩展分析。以Common Crawl和GAW为例，发现成对包含关系与发现曲线投影不一致，用双组件瓮模型调和，得出外壳不均匀，κ为秩解析泛化切入点。
AI中文摘要：
纵向网络爬虫是不断演变的URL群体的部分样本序列。两个爬虫之间的成对包含关系是标准探测方法；在一个简单的爬虫瓮模型下，每次轮次抽取一部分URL并替换一部分，可得到每轮生存率α和覆盖率c这两个可解释的比率，但该模型将群体视为均匀的且每次处理一对。在这项工作中，我们定义了一种用于谈论爬虫的形式语言。我们用发现曲线U(s,T)扩展了这一分析，它是从第s次开始的T次爬虫滑动窗口上的累积URL足迹，在相同瓮模型下也是(α,c)的封闭形式函数。包含关系和发现曲线是同一过程的两个投影：当瓮是均匀时，独立拟合在(α,c)上一致，所以任何不一致本身就是一种度量。应用于Common Crawl（2020 - 2025，域名粒度）和德国学术网络（GAW，URL粒度）时，两个投影在两个档案上都不一致，一个具有持久核心分数κ以及外壳参数(α∂,c∂)的双组件瓮模型调和了这种不一致。c∂上仍有残差，表明外壳本身不均匀；κ被记录为秩解析泛化的标量切入点，后续工作将继续跟进。
英文摘要：
A longitudinal web crawl is a sequence of partial samples of an evolving URL population. Pairwise containment between two crawls is the standard probe; under a simple \emph{urn} model of the crawl -- each round samples a fraction of the URLs and replaces a fraction -- it recovers two interpretable rates, per-round survival $α$ and coverage $c$, but treats the population as uniform and consumes one pair at a time. In this work, we define a formal language for talking about a crawl. We extend this analysis with the \emph{discovery curve} $U(s, T)$, the cumulative URL footprint over a sliding window of $T$ crawls starting at $s$, which under the same urn model is also a closed-form function of $(α, c)$. Containment and the discovery curve are then two projections of one process: independent fits agree on $(α, c)$ when the urn is homogeneous, so any disagreement is itself a measurement. Applied to Common Crawl (2020--2025, domain granularity) and to the German Academic Web (GAW, URL granularity), the two projections disagree on both archives, and a two-component urn with a persistent core fraction $κ$ alongside shell parameters $(α_\partial, c_\partial)$ reconciles the disagreement. A residual on $c_\partial$ remains, signaling that the shell itself is not homogeneous; $κ$ is recorded as the scalar entry point to a rank-resolved generalization, which is left to follow-up work. \keywords{web archive \and crawl coverage \and discovery curve \and urn model \and two-component model \and URL lifetime}
