We present QUOKA: Query-oriented KV selection for efficient attention, a training-free and hardware agnostic sparse attention algorithm for accelerating transformer inference under chunked prefill. While many queries focus on a smaller group of keys in the attention operator, we observe that queries with low cosine similarity with respect to the mean query interact more strongly with more keys and have the greatest contribution to final attention logits. By prioritizing these low cosine similarity queries, the behavior of full attention during the prefill stage can be closely approximated. QUOKA leverages this observation, accelerating attention by (1) first retaining a small set of representative queries and (2) then subselectin the keys most aligned with those queries. Through experiments on Needle-In-A-Haystack, LongBench, RULER, and Math500, we show that, while realizing a 3x reduction in time-to-first-token, 5x speedup in attention on Nvidia GPUs and up to nearly a 7x speedup on Intel Xeon CPUs, QUOKA achieves near-baseline accuracy, utilizing 88% fewer key-value pairs per attention evaluation.
QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving
Zhejiang University(浙江大学);East China Normal University(华东师范大学);Ant Group(蚂蚁集团);The Hong Kong Polytechnic University(香港理工大学);Zhejiang Normal University(浙江师范大学);Tongji University(同济大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学)
MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术产业开发区(滨江)区块链与数据安全研究院)
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Limerick(利默里克大学);CUHK, Shenzhen(香港中文大学(深圳))
ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models
Department of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术系);National Key Laboratory of Smart Farm Technologies and Systems(国家智能农业技术与系统重点实验室);Department of Computer Science and Technology, Harbin Institute of Technology (ShenZhen)(哈尔滨工业大学(深圳)计算机科学与技术系)
Middle Tennessee State University(中田纳西州立大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Tsinghua University(清华大学);The Simons Institute for the Theory of Computing at the UC, Berkeley(加州大学伯克利分校西蒙斯理论计算研究所)
Efficient Low Rank Attention for Long-Context Inference in Large Language Models
Guangdong University of Technology(广东工业大学);RIKEN AIP(理化学研究所AIP);Key Laboratory of Intelligent Detection and the Internet of Things in Manufacturing, Ministry of Education, Guangzhou, CHINA(教育部智能制造智能化检测与物联网重点实验室);RIKEN iTHEMS(理化学研究所iTHEMS);RIKEN IMS(理化学研究所IMS);Chiba University(千叶大学)
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Baidu Inc.(百度公司)
DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
Leiden Institute of Advanced Computer Science (LIACS), Leiden University(莱顿先进计算机科学研究所(LIACS)、莱顿大学);Department of Microelectronics, Delft University of Technology(电子系、代尔夫特理工大学)
School of Information, Renmin University of China(中国人民大学信息学院);Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China(数据工程与知识工程重点实验室);Engineering Research Center of Database and Business Intelligence, Beijing, China(数据库与商业智能工程研究中心);Ant Group, Hangzhou, China(蚂蚁集团)
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究所);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Tencent Youtu Lab, Shanghai, China(腾讯云智实验室,上海,中国)
Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AI Lab, The Yangtze River Delta(长江三角洲人工智能实验室);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University College London(伦敦大学学院);The Hong Kong University of Science and Technology(香港科学与技术大学);UCL Centre for Artificial Intelligence(伦敦大学学院人工智能中心)