Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.
MelTok: 2D Tokenization for Single-Codebook Audio Compression
International Digital Economy Academy (IDEA)(国际数字经济学院);Chinese University of Hong Kong(香港中文大学);Shenzhen Graduate School, Peking University(北京大学深圳研究生院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生学院)
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机学院,北京大学);Basic Model Technology Center, WeChat AI, Tencent Inc.(基础模型技术中心,微信AI,腾讯公司)
Concordia University(康科迪亚大学);Mila-Quebec AI Institute(魁北克人工智能研究所);The Hebrew University of Jerusalem(特拉维夫大学);University of Cambridge(剑桥大学);Indiana University(印第安纳大学);Carnegie Mellon University(卡内基梅隆大学);Microsoft(微软公司);Université de Montréal(蒙特利尔大学);Université de Toulon(托尔努瓦大学);Google(谷歌公司);Apple(苹果公司);Laval University(拉瓦尔大学);National Taiwan University(台湾大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
Carnegie Mellon University(卡内基梅隆大学);Amazon GenAI(亚马逊生成人工智能);James Silberrad Brown Center for Artificial Intelligence(詹姆斯·西伯拉德·布朗人工智能中心);University of Bristol(布里斯托大学);Stanford University(斯坦福大学);Northeastern University(东北大学);New York University(纽约大学)
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所);NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)(国立清华大学人工智能研究中心);ASUS Intelligent Cloud Services(ASUS智能云服务)
Token-Based Audio Inpainting via Discrete Diffusion
School of Electrical and Computer Engineering, Ben-Gurion University of the Negev(本·古里安大学电子与计算机工程学院);Faculty of Computer and Information Science, Data Science Research Center, Ben-Gurion University of the Negev(本·古里安大学计算机与信息科学学院);University of Haifa(海法大学)