We present a geometric framework for analysing multi-head attention in large language models (LLMs). Without altering the mechanism, we view standard attention through a top-N selection lens and study its behaviour directly in value-state space. We define geometric metrics - Precision, Recall, and F-score - to quantify separability between selected and non-selected tokens, and derive non-asymptotic bounds with explicit dependence on dimension and margin under empirically motivated assumptions (stable value norms with a compressed sink token, exponential similarity decay, and piecewise attention weight profiles). The theory predicts a small-N operating regime of strongest non-trivial separability and clarifies how sequence length and sink similarity shape the metrics. Empirically, across LLaMA-2-7B, Gemma-7B, and Mistral-7B, measurements closely track the theoretical envelopes: top-N selection sharpens separability, sink similarity correlates with Recall. We also found that in LLaMA-2-7B heads specialize into three regimes - Retriever, Mixer, Reset - with distinct geometric signatures. Overall, attention behaves as a structured geometric classifier with measurable criteria for token selection, offering head level interpretability and informing geometry-aware sparsification and design of attention in LLMs.
Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Advanced Micro Devices, Inc., Beijing, China（Advanced Micro Devices公司，北京）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Microsoft（微软公司）；Rutgers University（罗格斯大学）；California Institute of Technology（加州理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Identifying Semantic Induction Heads to Understand In-Context Learning
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Institute of Natural Sciences, Shanghai Jiao Tong University（上海交通大学自然科学研究院）
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Alibaba Cloud Computing（阿里云计算）；Xiaohongshu Inc.（小红书公司）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
RecurFormer: Not All Transformer Heads Need Self-Attention
Nanyang Technological University（南洋理工大学）；The University of New South Wales（新南威尔士大学）；Shanghai Jiao Tong University（上海交通大学）；The College of William and Mary（威廉玛丽学院）
Pair-In, Pair-Out: Latent Multi-Token Prediction for Efficient LLMs
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学首都人工智能学院）；AI Platform, Xiaohongshu Inc.（小红书人工智能平台）；University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学）；Tencent Youtu Lab（腾讯优图实验室）
E2S-Pruner: Progressive Two-Stage Evidence Fusion for Visual Token Pruning in Vision-Language Models
Digital Environment Research Institute (DERI), Queen Mary University of London（伦敦大学玛丽女王学院数字环境研究所（DERI））；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（华东师范大学上海人工智能教育研究院）；National Institute of Education, Nanyang Technological University（南洋理工大学国家教育学院）；School of Computer Science and Engineering, Jiangsu University of Science and Technology（江苏科技大学计算机科学与工程学院）
Dynamic K-Head: Difficulty-Supervised Budget Selection for Continuous Latent Reasoning
Beijing Institute of Mathematical Sciences and Applications（北京数学科学与应用研究院）；Tsinghua University（清华大学）；Peking University（北京大学）；The University of Hong Kong（香港大学）
M2Tok: Multi-head Multi-codebook Discrete Action Tokenization for Vision-Language-Action Models
The Hong Kong Polytechnic University（香港理工大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）
