The quadratic complexity of attention remains the central bottleneck in long-context inference for large language models. Prior acceleration methods either sparsify the attention map with structured patterns or permanently evict tokens at specific layers, which can retain irrelevant tokens or rely on irreversible early decisions despite the layer-/head-wise dynamics of token importance. In this paper, we propose Token Sparse Attention, a lightweight and dynamic token-level sparsification mechanism that compresses per-head $Q$, $K$, $V$ to a reduced token set during attention and then decompresses the output back to the original sequence, enabling token information to be reconsidered in subsequent layers. Furthermore, Token Sparse Attention exposes a new design point at the intersection of token selection and sparse attention. Our approach is fully compatible with dense attention implementations, including Flash Attention, and can be seamlessly composed with existing sparse attention kernels. Experimental results show that Token Sparse Attention consistently improves accuracy-latency trade-off, achieving up to $\times$3.23 attention speedup at 128K context with less than 1% accuracy degradation. These results demonstrate that dynamic and interleaved token-level sparsification is a complementary and effective strategy for scalable long-context inference.
University of Campinas（坎皮纳斯大学）；University of Glasgow（格拉斯哥大学）；NeuralMind
The ability to extrapolate, i.e., to make predictions on sequences that are longer than those presented as training examples, is a challenging problem for current deep learning models. Recent work shows that this limitation persists in state-of-the-art Transformer-based models. Most solutions to this problem use specific architectures or training methods that do not generalize to other tasks. We demonstrate that large language models can succeed in extrapolation without modifying their architecture or training procedure. Our experimental results show that generating step-by-step rationales and…
Beihang University（北航）；M-A-P；The Hong Kong Polytechnic University（香港理工大学）；AIWaves；University of Alberta（阿尔伯塔大学）；University of Waterloo（滑铁卢大学）；University of Manchester（曼彻斯特大学）；Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Shanghai AI Lab（上海AI实验室）；Nanjing University（南京大学）；Kuaishou Technology（快手科技）
An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM
The Graduate Center, CUNY（CUNY研究生中心）；Brooklyn College, CUNY（CUNY布鲁克林学院）；Borough of Manhattan Community College, CUNY（CUNY曼哈顿社区学院）；The City College of New York, CUNY（CUNY纽约城市学院）
Interleaved Latent Visual Reasoning with Selective Perceptual Modeling
China University of Geosciences, Wuhan（中国地质大学（武汉））；Shanghai Innovation Institute（上海创新研究院）；University of Southern California（南加州大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）
UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；Zhongguancun Academy（中关村学院）
AdaptR1: Reinforcement Learning Based Adaptive Interleaved Thinking in Multi-hop Question Answering
Computer Science, Fudan University（复旦大学计算机科学系）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究院）；Shanghai Innovation Institute（上海创新研究院）；Soochow University（苏州大学）
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
Aleph Alpha；Technical University Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Virginia Tech（弗吉尼亚理工大学）；Intuit AI Research（Intuit AI研究院）；Meta AI；The Chinese University of Hong Kong（香港中文大学）；University of California, Davis（加州大学戴维斯分校）
Facebook AI Research（Facebook人工智能研究院）；University of Washington（华盛顿大学）
We introduce a new balanced assignment of experts (BASE) layer for large language models that greatly simplifies existing high capacity sparse layers. Sparse layers can dramatically improve the efficiency of training and inference by routing each token to specialized expert modules that contain only a small fraction of the model parameters. However, it can be difficult to learn balanced routing functions that make full use of the available experts; existing approaches typically use routing heuristics or auxiliary expert-balancing loss functions. In contrast, we formulate token-to-expert alloca…
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；ILLC, University of Amsterdam（阿姆斯特丹大学语言、逻辑与计算研究所（ILLC））；Department of Computational Linguistics, University of Zurich（苏黎世大学计算语言学系）
Independence assumptions during sequence generation can speed up inference, but parallel generation of highly inter-dependent tokens comes at a cost in quality. Instead of assuming independence between neighbouring tokens (semi-autoregressive decoding, SA), we take inspiration from bidirectional sequence generation and introduce a decoder that generates target words from the left-to-right and right-to-left directions simultaneously. We show that we can easily convert a standard architecture for unidirectional decoding into a bidirectional decoder by simply interleaving the two directions and a…
Ontology Memory-Augmented ASR Correction for Long Text-Speech Interleaved Conversations
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), China（哈尔滨工业大学（深圳）计算与智能研究所）；Shenzhen Loop Area Institute (SLAI), China（深圳环域研究所）
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft STC Asia（微软亚太研发集团）
Transformer has achieved great success in NLP. However, the quadratic complexity of the self-attention mechanism in Transformer makes it inefficient in handling long sequences. Many existing works explore to accelerate Transformers by computing sparse self-attention instead of a dense one, which usually attends to tokens at certain positions or randomly selected tokens. However, manually selected or random tokens may be uninformative for context modeling. In this paper, we propose Smart Bird, which is an efficient and effective Transformer with learnable sparse attention. In Smart Bird, we fir…
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Google Research（谷歌研究院）
Multi-vector retrieval models improve over single-vector dual encoders on many information retrieval tasks. In this paper, we cast the multi-vector retrieval problem as sparse alignment between query and document tokens. We propose AligneR, a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens (e.g. `dog' vs. `puppy') and per-token unary saliences reflecting their relative importance for retrieval. We show that controlling the sparsity of pairwise token alignments often brings significant performance gains. While most factoid question…
