In spite of the great potential of large language models (LLMs) across various tasks, their deployment on resource-constrained devices remains challenging due to their excessive computational and memory demands. Quantization has emerged as an effective solution by storing weights in reduced precision. However, utilizing low precisions (i.e.~2/3-bit) to substantially alleviate the memory-boundedness of LLM decoding, still suffers from prohibitive performance drop. In this work, we argue that existing approaches fail to explore the diversity in computational patterns, redundancy, and sensitivity to approximations of the different phases of LLM inference, resorting to a uniform quantization policy throughout. Instead, we propose a novel phase-aware method that selectively allocates precision during different phases of LLM inference, achieving both strong context extraction during prefill and efficient memory bandwidth utilization during decoding. To further address the memory-boundedness of the decoding phase, we introduce Progressive Mixed-Precision Decoding (PMPD), a technique that enables the gradual lowering of precision deeper in the generated sequence, together with a spectrum of precision-switching schedulers that dynamically drive the precision-lowering decisions in either task-adaptive or prompt-adaptive manner. Extensive evaluation across diverse language tasks shows that when targeting Nvidia GPUs, PMPD achieves 1.4$-$12.2$\times$ speedup in matrix-vector multiplications over fp16 models, while when targeting an LLM-optimized NPU, our approach delivers a throughput gain of 3.8$-$8.0$\times$ over fp16 models and up to 1.54$\times$ over uniform quantization approaches while preserving the output quality.
Integrative Decoding: Improve Factuality via Implicit Self-consistency
The Hong Kong Polytechnic University(香港理工大学);Tsinghua University(清华大学);Microsoft Research(微软研究院);Microsoft Azure AI(微软Azure人工智能);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Self Speculative Decoding for Diffusion Large Language Models
School of AI, Shanghai Jiao Tong University(上海交通大学人工智能学院);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学);Shanghai AI Laboratory(上海人工智能实验室);Huawei Technologies Ltd.(华为技术有限公司)
Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion
King Abdullah University of Science and Technology(国王阿卜杜勒阿齐兹大学科学与技术大学);Shenzhen International Center for Industrial and Applied Mathematics, Shenzhen Research Institute of Big Data(深圳国际工业与应用数学中心,深圳大数据研究院);Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong(香港中文大学);King Abdulaziz University(国王阿卜杜勒阿齐兹大学);Prince Sattam bin Abdulaziz University(普林斯萨塔姆·本·阿卜杜勒阿齐兹大学);University of Bisha(比沙大学)
DeepHQ: Learned Hierarchical Quantizer for Progressive Deep Image Coding
Electronics and Telecommunications Research Institute(电信与信息通信研究院);Korea Advanced Institute of Science and Technology(韩国科学技术院);Telecommunications Research Institute(电信研究 institute)
SHSP: Structure-Aware Hierarchical Solution Prediction for Mixed-Integer Linear Programming
State Key Laboratory of Novel Software Technology, Nanjing University(南京大学 现代软件工程国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学 人工智能学院);State Key Laboratory of Technology and Equipment for Defense Against Power System Operational Risks, Nari Technology Co., Ltd.(南瑞科技股份有限公司 电力系统运行风险防御技术与装备国家重点实验室)
Iterative Encoding-Decoding VAEs Anomaly Detection in NOAA's DART Time Series: A Machine Learning Approach for Enhancing Data Integrity for NASA's GRACE-FO Verification and Validation
University of California at Los Angeles(加利福尼亚大学洛杉矶分校);NASA Jet Propulsion Laboratory(美国国家航空航天局喷气推进实验室)