In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable. However, caching transformer states can easily require almost as much space as the model parameters. When the right context isn't known in advance, caching ICL can be challenging. This work addresses these limitations by introducing models that, inspired by the encoder-decoder architecture, use cross-attention to condition generation on reference text without the prompt. More precisely, we leverage pre-trained decoder-only models and only train a small number of added layers. We use Question-Answering (QA) as a testbed to evaluate the ability of our models to perform conditional generation and observe that they outperform ICL, are comparable to fine-tuned prompted LLMs, and drastically reduce the space footprint relative to standard KV caching by two orders of magnitude.
LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs
Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））；Shanghai Jiao Tong University（上海交通大学）；University at Buffalo（纽约州立大学布法罗分校）
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Technical University of Munich（慕尼黑技术大学）；Technical University of Darmstadt（达姆施塔特技术大学）；Zhejiang University（浙江大学）；Technische Universität Ilmenau（伊尔梅瑙技术大学）
Can Transformer Memory Be Corrupted? Investigating Cache-Side Vulnerabilities in Large Language Models
College of Engineering and Computer Science, University of Central Florida（中央佛罗里达大学工程与计算机科学学院）；Department of Computer Science and Engineering, Mississippi State University（密苏里州立大学计算机科学与工程系）；Department of Electrical and Computer Engineering, North Carolina State University（北卡罗来纳州立大学电子与计算机工程系）
