zip2zip++：面向推理时的最小质量损失的令牌压缩
zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss
AI总结：
zip2zip++通过改进压缩层与LZW编解码器，解决了动态词汇表压缩的质量损失问题，在不修改预训练语言模型的情况下，缩小了zip2zip与未压缩基线的质量差距，提升了困惑度与数学推理准确率。
摘要：
基于Lempel-Ziv-Welch（LZW）压缩动态构建的词汇表可让语言模型用更少的令牌表示相同文本，但现有研究表明这种效率提升会以牺牲推理能力和困惑度为代价。本文证明这种质量差距并非压缩的固有代价，而是可以通过对压缩层和LZW编解码器进行针对性改进来大幅缩小，且无需修改底层预训练语言模型。我们的分析揭示了四个关键问题：（1）LZW压缩算法在输入嵌入和输出嵌入之间产生隐式冲突，单个共享嵌入无法解决这种张力；（2）将RoPE直接应用于LZW压缩序列时，忽略了合并后的令牌可能跨越多个令牌；（3）合并某些令牌（尤其是数字）会不成比例地损害下游任务准确率；（4）动态词汇表引入了固有的分割歧义，会干扰标准评估。我们通过解耦的输入/输出动态嵌入层、感知大小的RoPE定位，以及支持选择性合并和多视图分割的改进型LZW编解码器来解决这些问题。在四个模型上，我们的方案在所有评估的语料库上均提升了困惑度，同时也提升了数学推理准确率，在通用文本的相似压缩率下，将zip2zip与未压缩基线之间的质量差距缩小了约一半。这代表着向语言模型的无损自适应词汇表压缩迈出了重要一步。
英文摘要：
Dynamic vocabularies built on-the-fly via Lempel-Ziv-Welch (LZW) compression let a language model represent the same text in fewer tokens, but prior work shows that this efficiency comes at a cost to reasoning and perplexity. We show that this quality gap is not a fundamental price of compression but can be substantially closed through targeted improvements to the compression layer and LZW codec, without modifying the underlying pretrained language model. Our analysis reveals four key issues: (1) LZW's compression algorithm creates an implicit conflict between the input embedding and the output embedding, a tension that a single shared embedding cannot resolve; (2) naively applying RoPE to LZW-compressed sequences ignores that merged tokens can span multiple tokens; (3) merging certain tokens (notably digits) disproportionately hurts downstream accuracy; and (4) dynamic vocabularies introduce an inherent segmentation ambiguity that confounds standard evaluation. We address these through decoupled input/output dynamic embedding layer, size-aware RoPE positioning, and a modified LZW codec supporting selective merges and multi-view segmentations. Across four models, our recipe improves perplexity on every corpus we evaluated as well as mathematical reasoning accuracy, closing roughly half the quality gap between zip2zip and the uncompressed baseline at a similar compression ratio on general text. This represents a significant step toward lossless adaptive-vocabulary compression for language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
