Delta Tokenizer：面向多核CPU的会话增量参考一致分词与分段并行长提示词处理
Delta Tokenizer: Reference-Identical Tokenization of Session Deltas and Segment-Parallelism for Long Prompts on Multicore CPUs
AI总结：
Delta Tokenizer是面向多核CPU的分词前端，通过增量分词与分段并行技术，大幅缩短会话轮次及长请求的分词时间，降低LLM服务的首令牌生成时间，且输出与参考结果完全一致。
摘要：
驱动大语言模型（LLM）的智能体会反复向模型发送相同对话，每一轮仅在末尾添加少量文本。推理引擎会复用共享前缀的键值（KV）缓存，因此仅将新令牌传入模型，但分词过程无此类捷径。服务引擎的CPU前端会将每个请求从首个字符转换回令牌ID，这会延长首令牌生成时间（TTFT）并在每一轮消耗CPU资源。本文提出Delta Tokenizer，一种CPU分词前端，仅对请求与同一会话前一请求之间的增量部分进行分词。它保留早期令牌序列，仅重新编码附加文本及附加内容可能影响的短尾部，且在分词规则自身无法移动的位置将两部分拼接，以保持结果精确。对于无可复用状态的长请求，它会在实用分词器已保持相邻片段独立的位置切割文本，同时在多个核心上并行编码这些片段，并将输出拼接为参考编码，结果完全一致。共享CPU池会将这些片段与短增量更新一同调度，使更新在两个片段之间运行，而非在整个长任务之后。增量路径相比Hugging Face参考分词器，可将会话轮次的分词速度缩短约40倍；在16核上，片段并行路径对400万字符请求的编码速度比串行编码快8.2倍，比LoPT快1.6至2.1倍。在SGLang内部，Delta Tokenizer在10万令牌时将聊天界面的中位TTFT降低59%，在负载下将智能体会话的P99 TTFT降低9.5倍，在长请求占用前端时降低23倍。经检查，其所有输出均与参考结果匹配，包括其快速编码器编码的418亿个文档。
英文摘要：
An agent that drives an LLM sends the model the same conversation again and again, and each turn adds only a little text at the end. Inference engines reuse the KV cache of the shared prefix, so only the new tokens pass through the model. However, tokenization has no such shortcut. The CPU front end of the serving engine converts every request back into token IDs from its first character, which lengthens the time to first token (TTFT) and consumes CPU on every turn. We present Delta Tokenizer, a CPU tokenization front end that tokenizes only the delta between a request and the previous request of the same session. It retains the earlier token sequence, encodes again only the appended text and the short tail whose tokens the append could affect, and joins the two only at a position the tokenizer's splitting rules themselves cannot move, which keeps the result exact. For long requests with no state to reuse, it cuts the text where practical tokenizers already keep neighboring pieces independent, encodes the pieces on several cores at once, and concatenates the outputs into the reference encoding, exactly. A shared CPU pool schedules these chunks together with the short incremental updates, so an update runs between two chunks rather than behind an entire long job. The incremental path shortens the tokenization of a session turn by roughly 40x relative to the Hugging Face reference tokenizer, and on 16 cores the chunk-parallel path encodes a four-million-character request 8.2x faster than a serial encode and 1.6-2.1x faster than LoPT. Inside SGLang, Delta Tokenizer reduces the median TTFT through the chat interface by 59% at 100K tokens and brings the P99 TTFT of agent sessions down 9.5x under load and 23x when long requests tie up the front end. Every output we checked matches the reference, among them 41.8 billion documents encoded by its fast encoder.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
