The high inference cost of Large Language Models (LLMs) poses challenges, especially for tasks requiring lengthy outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language (i.e., concise outputs that retain essential meaning) when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which the LLM omits a predefined set of semantically irrelevant and easily inferable words based on the context during inference. Then, a specifically trained smaller language model with lower inference cost reconstructs the distilled answer into the ideal answer. Our experiments show promising results, particularly on the proposed NaLDA evaluation dataset focused on the reconstruction task, with 19.4% saved tokens on average for GPT-4o and only a tiny decrease in evaluation metrics. This suggests that the approach can effectively balance efficiency and accuracy in language processing tasks.
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tencent Inc.(腾讯公司);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院)
Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
School of Physics, Peking University(北京大学物理学院);Institute of Theoretical Physics, Chinese Academy of Sciences(中国科学院理论物理研究所);Center for High Energy Physics, Peking University(北京大学高能物理研究中心);Beijing Computational Science Research Center(北京计算科学研究中心)
TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction
Meetyou AI Lab(美柚人工智能实验室);Xiamen Key Laboratory of Women’s Internet Health Management(厦门市女性互联网健康管理重点实验室);Osaka University(大阪大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局)
Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteria
POSTECH;Seoul National University(首尔国立大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Kyungpook National University(庆北国立大学);LG AI Research(LG AI研究院)
Birkbeck, University of London(伦敦大学伯贝克学院);School of Advanced Study, Institute of Philosophy, University of London(伦敦大学高等研究院哲学研究所);University of Utah(犹他大学);Tufts University(塔夫茨大学)
Lossless Vocabulary Reduction for Auto-Regressive Language Models
NTT Computer and Data Science Laboratories, NTT Corporation(NTT计算机与数据科学实验室,NTT集团);NTT Human Informatics Laboratories, NTT Corporation(NTT人类信息学实验室,NTT集团)
HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit
Institute of Digital Twin, Eastern Institute of Technology(数字孪生研究所,东部技术研究所);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Munich Center for Machine Learning, LMU Munich(慕尼黑大学机器学习中心,慕尼黑大学)