Attention in Constant Time: Vashista Sparse Attention for Long-Context Decoding with Exponential Guarantees
Datar Consulting(达特咨询)
Vashista Nobaub
备注:
The author withdraws this paper because the proof of Theorem 1 relies on an incorrect claim in Appendix A.5 (Lemma 3). The lemma claims that the regularized solution stays on the original face of the convex hull, but the regularization gives positive weight to points outside that face.
Large language models spend most of their inference cost on attention over long contexts, yet empirical behavior suggests that only a small subset of tokens meaningfully contributes to each query. We formalize this phenomenon by modeling attention as a projection onto the convex hull of key vectors and analyzing its entropic (softmax-like) relaxation. Our main theoretical contribution is a face-stability theorem showing that, under a strict complementarity margin (a support gap (Δ) certified by KKT multipliers), entropic attention concentrates on a constant-size active face: the total mass assigned to inactive tokens decays exponentially as (\exp(-Ω(Δ/\varepsilon))), while the error on the active face scales linearly in the temperature/regularization parameter (\varepsilon). This yields a practical criterion for when sparse long-context decoding is safe and provides a principled knob to trade accuracy for compute.
Building on these guarantees, we introduce Vashista Sparse Attention, a drop-in mechanism that maintains a small candidate set per query through a paging-style context selection strategy compatible with modern inference stacks. Across long-context evaluations, we observe stable constant-size effective support, strong wall-clock speedups, and minimal quality degradation in the regimes predicted by the support-gap diagnostics. Finally, we discuss deployment implications for privacy-sensitive and air-gapped settings, where interchangeable attention modules enable predictable latency and cost without external retrieval dependencies.
CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
Department of Computer Science, Yale University(耶鲁大学计算机科学系);College of Engineering, Columbia University(哥伦比亚大学工程学院);Department of Statistics, University of Wisconsin-Madison(威斯康星大学麦迪逊分校统计学系)
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta(阿尔伯塔大学计算机科学系与阿尔伯塔机器智能研究所);Canada CIFAR AI Chair(加拿大CIFAR人工智能讲席)
Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning
Institute of Statistics and Big Data, Renmin University of China(中国人民大学统计与大数据研究院);Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China(中国人民大学大数据与国家治理研究院);Center for Applied Statistics, Institute of Statistics and Big Data, Renmin University of China(中国人民大学统计与大数据研究院应用统计中心)
Department of XXX, University of YYY, Location, Country(XXX系,YYY大学,地点,国家);School of ZZZ, Institute of WWW, Location, Country(ZZZ学院,WWW研究所,地点,国家);JP Morgan AI Research, London, UK(摩根大通AI研究,伦敦,英国);JP Morgan AI Research, New York, USA(摩根大通AI研究,纽约,美国)