认证切分：最大吞噬分词的可判定性与锚点供给
Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization
浏览论文内容
中文总结 AI 辅助
本文提出认证切分方法，通过判定令牌集合的认证位置并生成锚点清单，实现无需推测或修正的独立块扫描，并给出编辑边界移动的定理及差分审计工具。
中文摘要 AI 辅助
认证（certificate）是一个字节窗口，其内部包含一个起点（origin）：无论该窗口出现在完全可分词（completely tokenizable）的输入中的何处，覆盖窗口最后一个字节的令牌（token）均从该起点开始。仅根据令牌集合即可判定，一个认证位置（certified position）标记了可以安全开始工作的位置。本文判定认证问题，并将其锚点（anchors）向前读取，应用于切分（cuts）、编辑（edits）和审计（audits）。认证以两种方式判定：在线方式针对字面词汇表（literal vocabularies）使用完整性截断（completeness cutoff），离线方式通过武装运行验证器（armed-run verifier）处理任意正则令牌集合。无论采用哪种路径，在声明的窗口预算（window budget）下，均能生成锚点清单（anchor inventory），并对每次拒绝提供见证（witness）。其结果是若干定理。在认证锚点处切分输入并独立扫描各块，可重现顺序分词（sequential segmentation），无需推测（speculation）或修正过程（fixup pass）。若字节替换保持输入可分词，则边界移动严格限制在替换两侧未变字节中已见证的认证锚点之间。对于长度至多为L的令牌，位置p处的编辑不会移动位于p-L+1或更下方的任何边界。一个分隔符（delimiter）仅当其仅出现在令牌开头（token-initial）时，才允许在其之前进行安全切分；仅当其仅出现在令牌末尾（token-final）时，才允许在其之后进行安全切分，因此嵌入分隔符（embedded-delimiter）的失败会被拒绝。随之而来的是两种工具：有限认证清单的无锚跨度（anchor-free span），判定其有界与否并给出精确上确界（exact supremum），对于UTF-8形状则为三；以及一个差分审计器（differential auditor），判定两个令牌集合是否在两者均可分词的输入上放置不同边界，并提供见证。每个定理都带有可执行的检查。每项测量都是命名程序的输出或基于此类输出的算术运算，每个程序随论文附带或通过提交（commit）固定在所研究库中；该库的线程化扫描器（threaded scanner）在认证锚点处切分存档的活动语料库（campaign corpus），其边界流（boundary stream）在字节层面完全一致。
英文摘要
A certificate is a window of bytes with an origin inside it: wherever the window occurs in a completely tokenizable input, the token covering the window's final byte begins at that origin. Decided from the token set alone, a certified position marks where work may soundly begin. This paper decides certification and reads its anchors forward, into cuts, edits and audits. Certification is decided two ways: online with a completeness cutoff for literal vocabularies, and offline through an armed-run verifier for arbitrary regular token sets. Either route, under a declared window budget, yields the anchor inventory, with a witness on each refusal. The consequences are theorems. Cutting an input at certified anchors and scanning the chunks independently reproduces the sequential segmentation, with no speculation and no fixup pass. A byte substitution that keeps the input tokenizable moves boundaries only strictly between the certified anchors witnessed in the unchanged bytes on either side of it. For tokens of length at most L, an edit at p moves no boundary at or below p - L + 1. A delimiter permits a sound cut before it exactly when it sits only token-initial, and one after it exactly when only token-final, so the embedded-delimiter failure is refused. Two instruments follow: the anchor-free span of a finite certified inventory, decided bounded or not with its exact supremum, three for the UTF-8 shape; and a differential auditor deciding whether two token sets place different boundaries on an input both tokenize, with witnesses. Every theorem carries an executable check. Every measurement is a named program's output or arithmetic on such outputs, each program shipped with the paper or pinned by commit in the library it studies; that library's threaded scanner splits the archived campaign corpus at certified anchors, its boundary stream byte-identical throughout.
