TLDR：压缩音频令牌以实现高效自回归文本到语音
TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
AI总结：
提出TLDR框架，通过将因果建模从令牌级转移到补丁级，利用轻量级压缩器和LoRA适配的冻结预训练骨干，实现1.8倍推理加速和75% KV缓存减少。
摘要：
基于编解码器的自回归（AR）语音语言模型通过将语音建模为离散音频令牌序列，并使用大型预训练骨干网络，实现了强大的文本到语音（TTS）质量。然而，这种令牌级公式造成了结构效率瓶颈：语音令牌序列比文本序列长得多，要求AR骨干在每个令牌位置执行因果计算，并维护随序列长度增长的KV缓存。我们引入TLDR，一种基于补丁的自回归框架，通过将因果建模从令牌级语音序列转移到补丁级序列，加速基于编解码器的AR-TTS。TLDR使用轻量级压缩器将连续的编解码器令牌分组为紧凑的潜在补丁，使用通过LoRA适配的冻结预训练AR-TTS骨干对生成的较短补丁序列进行建模，并使用说话人条件提取器在每个补丁内重建细粒度语音令牌。在补丁大小为4的情况下，TLDR比基线AR-TTS模型实现了1.8倍的推理加速，并将全局KV缓存内存减少了高达75%。实验结果表明，补丁级全局因果建模可以成为降低预训练基于编解码器的AR-TTS系统推理成本的一种实用方法，而无需替换现有模块。
英文摘要：
Codec-based autoregressive (AR) speech language models have achieved strong text-to-speech (TTS) quality by modeling speech as sequences of discrete audio tokens with large pretrained backbones. However, this token-level formulation creates a structural efficiency bottleneck: speech-token sequences are much longer than text sequences, requiring the AR backbone to perform causal computation at every token position and maintain a KV cache that grows with the sequence length. We introduce TLDR, a patch-based autoregressive framework that accelerates codec-based AR-TTS by shifting the causal modeling from token-level speech sequences to patch-level sequences. TLDR groups consecutive codec tokens into compact latent patches using a lightweight compressor, models the resulting shorter patch sequence with a frozen pretrained AR-TTS backbone adapted by LoRA, and reconstructs fine-grained speech tokens within each patch using a speaker-conditioned extractor. With a patch size of 4, TLDR achieves a 1.8x inference speedup over the baseline AR-TTS model and reduces global KV-cache memory by up to 75%. Experimental results indicate that patch-level global causal modeling can be a practical way to reduce the inference cost of pretrained codec-based AR-TTS systems without replacing the existing modules.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
