公告携带线索：标记、边界与预训练语料的表示法
The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora
浏览论文内容
中文总结 AI 辅助
该研究定义清洁窗口存活度，从三方面测量预训练语料的表示法，发现公告是关键线索，发布纯框架格式并提出数据卡片记录要求。
中文摘要 AI 辅助
文档的排列方式（即其表示法）是数据集卡片未记录的一个训练变量。该领域已证实，文本提取选择会改变模型行为，但从未对这些选择放入语料的内容的表示法进行测量。我们定义了“清洁窗口存活度”，这是一种确定性计数，用于衡量文本流中仍需边界推断的部分，并从三个方面对表示法进行测量：1. 语料库所承载的内容：对13个公共语料库的统计显示，在经视觉转换的PDF片段中，存活度降至0.153，而在C4中则为0.889；稀缺资源并非未标记文本，而是长未标记文本；一项预先注册的供应测试发现，剩余内容为机构性内容，而非消费性内容。我们自己预先注册的预测失败了：转换器不会在散文上编造结构，这一零结果迫使我们建立了能够应对该情况的可靠性机制。2. 读者所使用的内容：在跨越0.6B至8.2B的5个基础模型和两条流程中，删除结构性公告会使后续散文的可预测性显著降低，而替换其表示法则无任何影响。这一零结果并非意味着表示法不重要，而是将该变量重新定位：起作用的线索是公告，而非符号。3. 作者所施加的内容：有界空值。基础模型不会在作者基线之上施加标记语域，且在每篇公告被删除的散文中，它们不会补加任何公告，其概率与作者参考值（为零）相比无显著差异。我们发布了这些测量所暗示的格式：纯框架，按作者顺序排列的段落，所有公告被删除至一个可逆的辅助文件中，根据公告存在与否而非表示法，将其与标记副本混合。应根据训练的能力选择格式运算符，而非根据保留的保真度，并在数据卡片上记录提取器身份和存活度。
英文摘要
How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.
