We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios (e.g., retrieved documents). LLMA first selects a text span from the reference and copies its tokens to the decoder and then efficiently checks the tokens' appropriateness as the decoding result in parallel within one decoding step. The improved computational parallelism allows LLMA to achieve over 2x speed-up for LLMs with identical generation results as greedy decoding in many practical generation scenarios where significant overlap between in-context reference and outputs exists (e.g., search engines and multi-turn conversations).
Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, AAIS, Peking University（北京大学AAIS数据科学中心）；Meituan（美团）；Meta AI；National Key Laboratory of General Artificial Intelligence（通用人工智能全国重点实验室）；BIGAI（北京智源人工智能研究院）
To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
Dept. of Electrical and Computer Engineering, Rice University（电气与计算机工程系， Rice大学）；Dept. of Computer Science, Rice University（计算机科学系， Rice大学）；Lambda, Inc.（Lambda公司）；Dept. of Computer Science, Stevens Institute of Technology（计算机科学系， Stevens Institute of Technology）
Johns Hopkins University（约翰霍普金斯大学）；University of Alberta（阿尔伯塔大学）；University of British Columbia（不列颠哥伦比亚大学）；Avey Research Center（艾维研究中心）；Google DeepMind（谷歌DeepMind）；University of California（加州大学）
APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与技术学院）；Shanghai Qi Zhi Institute（上海启智研究院）；School of Integrated Circuits, Sun Yat-sen University（中山大学集成电路学院）
