F3-Tokenizer: 驯服音频自编码器潜在变量以支持理解与生成
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
AI总结：
针对连续音频自编码器潜在变量结构弱、自监督编码器不可解码的问题，提出F3-Tokenizer，通过噪声正则化自编码器瓶颈和潜在侧表示编码器，实现统一的理解与生成音频分词器。
摘要：
连续音频自编码器能很好地重建波形，但通常产生的潜在变量结构较弱，不利于理解；而自监督音频编码器能捕捉语义，但不可直接解码。这种不匹配使得单个音频分词器难以同时支持理解和生成。我们通过两个组件将连续自编码器潜在变量适应于这一场景：噪声正则化的自编码器瓶颈和潜在侧表示编码器。瓶颈使用通道归一化和随机扰动代替基于KL的变分训练，为重建和自回归生成提供尺度可控的连续潜在变量。表示编码器在冻结的自编码器潜在变量上使用RQ-MTP和冻结LLM监督进行训练。最终的分词器为理解提供高维表示，同时保留归一化的连续潜在变量作为生成目标。
英文摘要：
Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable. This mismatch complicates a single audio tokenizer that must support both understanding and generation. We adapt continuous autoencoder latents to this setting with two components: a noise-regularized autoencoder bottleneck and a latent-side representation encoder. The bottleneck uses channel normalization and stochastic perturbation instead of KL-based variational training, yielding scale-controlled continuous latents for reconstruction and autoregressive generation. The representation encoder is trained on frozen autoencoder latents with RQ-MTP and frozen-LLM supervision. The resulting tokenizer provides high-dimensional representations for understanding while preserving normalized continuous latents as generation targets
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
