Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a $2.6\times$ inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling.
High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification
Department of Statistics, Harvard University（哈佛大学统计学系）；Center of Mathematical Sciences and Applications, Harvard University（哈佛大学数学科学中心）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）
Natural language modeling with limited training data is a challenging problem, and many algorithms make use of large-scale pretrained language models (PLMs) for this due to its great generalization ability. Among them, additive learning that incorporates a task-specific adapter on top of the fixed large-scale PLM has been popularly used in the few-shot setting. However, this added adapter is still easy to disregard the knowledge of the PLM especially for few-shot natural language generation (NLG) since an entire sequence is usually generated by only the newly trained adapter. Therefore, in thi…
Selective Preference Optimization via Token-Level Reward Function Estimation
National Centre for Text Mining, The University of Manchester（曼彻斯特大学文本挖掘中心）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；Center for Language and Information Research, Wuhan University（武汉大学语言与信息研究中心）；The Fin AI（Fin AI公司）；Baidu Inc.（百度公司）；Archimedes Research（阿基米德研究）
Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
Southern University of Science and Technology（南方科技大学）；Peking University（北京大学）；Shanghai University of Finance and Economics（上海金融学院）；Tsinghua University（清华大学）
Think Again or Think Longer? Selective Verification for Budget-Aware Reasoning
Department of Computer Science, Virginia Tech（弗吉尼亚理工大学计算机科学系）；Fralin Biomedical Research Institute, Virginia Tech（弗吉尼亚理工大学弗拉林生物医学研究所）；FBRI Cancer Research Center（FBRI癌症研究中心）
A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification
Departamento de Computación, FCEyN Universidad de Buenos Aires（布宜诺斯艾利斯大学计算机系）；ELIAS Lab, Departamento de Ingeniería Universidad de San Andres（圣安德烈斯大学工程系ELIAS实验室）
SeLaR: Selective Latent Reasoning in Large Language Models
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology（广东省超高清沉浸式媒体技术重点实验室）；Shenzhen Graduate School, Peking University（北京大学深圳研究生院）
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；National University of Singapore（新加坡国立大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Models
MedVisAI Lab, Singapore（新加坡MedVisAI实验室）；Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, China（中国上海交通大学医学院瑞金医院）；Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore（新加坡南洋理工大学Lee Kong Chian医学院）
Beihang University（北京航空航天大学）；University of Montreal（蒙特利尔大学）；CENCERT/CC（国家计算机网络应急技术处理协调中心/国家互联网应急中心）
Hashtag generation aims to generate short and informal topical tags from a microblog post, in which tokens or phrases form the hashtags. These tokens or phrases may originate from primary fragmental textual pieces (e.g., segments) in the original text and are separated into different segments. However, conventional sequence-to-sequence generation methods are hard to filter out secondary information from different textual granularity and are not good at selecting crucial tokens. Thus, they are suboptimal in generating more condensed hashtags. In this work, we propose a modified Transformer-base…
SEER: Long-Context Reasoning via Selective Visual-Text Compression
The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Cambridge（剑桥大学）；University of Technology Sydney（悉尼科技大学）；The University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Token Level Routing Inference System for Edge Devices
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学（MBZUAI））；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Rutgers University（罗格斯大学）
Columbia University（哥伦比亚大学）；Virginia Tech（弗吉尼亚理工大学）；Google Research（谷歌研究院）
Protecting large language models from privacy leakage is becoming increasingly crucial with their wide adoption in real-world products. Yet applying differential privacy (DP), a canonical notion with provable privacy guarantees for machine learning models, to those models remains challenging due to the trade-off between model utility and privacy loss. Utilizing the fact that sensitive information in language data tends to be sparse, Shi et al. (2021) formalized a DP notion extension called Selective Differential Privacy (SDP) to protect only the sensitive tokens defined by a policy function. H…
