Retrieving documents and prepending them in-context at inference time improves performance of language model (LMs) on a wide range of tasks. However, these documents, often spanning hundreds of words, make inference substantially more expensive. We propose compressing the retrieved documents into textual summaries prior to in-context integration. This not only reduces the computational costs but also relieves the burden of LMs to identify relevant information in long retrieved documents. We present two compressors -- an extractive compressor which selects useful sentences from retrieved documents and an abstractive compressor which generates summaries by synthesizing information from multiple documents. Both compressors are trained to improve LMs' performance on end tasks when the generated summaries are prepended to the LMs' input, while keeping the summary concise.If the retrieved documents are irrelevant to the input or offer no additional information to LM, our compressor can return an empty string, implementing selective augmentation.We evaluate our approach on language modeling task and open domain question answering task. We achieve a compression rate of as low as 6% with minimal loss in performance for both tasks, significantly outperforming the off-the-shelf summarization models. We show that our compressors trained for one LM can transfer to other LMs on the language modeling task and provide summaries largely faithful to the retrieved documents.
Beyond Memorization: The Challenge of Random Memory Access in Language Models
Sea AI Lab（Sea AI实验室）；National University of Singapore（新加坡国立大学）；Carnegie Mellon University（卡内基梅隆大学）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Precise Information Control in Long-Form Text Generation
Paul G. Allen School of Computing Science & Engineering, University of Washington（保罗·G·艾伦计算科学与工程学院，华盛顿大学）；Princeton Language and Intelligence (PLI), Princeton University（普林斯顿语言与智能（PLI），普林斯顿大学）；Allen Institute for AI（艾伦人工智能研究所）
Neural cache language models (LMs) extend the idea of regular cache language models by making the cache probability dependent on the similarity between the current context and the context of the words in the cache. We make an extensive comparison of 'regular' cache models with neural cache models, both in terms of perplexity and WER after rescoring first-pass ASR results. Furthermore, we propose two extensions to this neural cache model that make use of the content value/information weight of the word: firstly, combining the cache probability and LM probability with an information-weighted int…
Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations
School of Data Science, The Chinese University of Hong Kong, Shenzhen（数据科学学院，香港中文大学（深圳））；School of Computer Science and Engineering (School of Cyber Security), University of Electronic Science and Technology of China（计算机科学与工程学院（网络安全学院），电子科学与技术大学）
Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
