基于组合适配器的上下文参数化
Context Parametrization with Compositional Adapters
AI总结：
CompAs通过组合结构的适配器参数，提升LLM在长上下文和多任务中的效率与稳定性，提供一种更高效的部署方案。
摘要：
大型语言模型（LLMs）通常通过上下文学习（ICL）或监督微调（SFT）无缝适应新任务。然而，当处理大量示例时，ICL效率低下，而SFT则带来训练开销并牺牲灵活性。将指令或示例直接映射到适配器参数是一种有吸引力的替代方案。尽管先前工作探索了基于单个输入上下文生成适配器，但忽略了整合多个信息块的需要。为解决这一缺口，我们引入CompAs，一种元学习框架，能够将上下文转换为具有组合结构的适配器参数，允许它们代数合并。这种方法带来了三个优势：较低的推理成本、在长上下文中稳定性提升，以及当输入超过模型上下文窗口时建立了一种原理性解决方案。此外，CompAs可逆地将信息编码到适配器参数中，使原始输入上下文得以恢复，从而促进安全性。在多样化的多项选择和提取性问答任务上的实验证明，CompAs优于ICL和先前的生成方法，尤其是在扩展输入规模时表现更佳。我们的工作确立了可组合的适配器生成作为扩展LLM部署的一种实用且高效的替代方案。
英文摘要：
Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, ICL is inefficient when handling many demonstrations, and SFT incurs training overhead while sacrificing flexibility. Mapping instructions or demonstrations from context directly into adapter parameters offers an appealing alternative. While prior work explored generating adapters based on a single input context, it has overlooked the need to integrate multiple chunks of information. To address this gap, we introduce CompAs, a meta-learning framework that translates context into adapter parameters with a compositional structure that allows them to be merged algebraically. This approach yields three benefits: lower inference cost, improved stability under long contexts, and establishes a principled solution when input exceeds the model's context window. Furthermore, CompAs reversibly encodes information into adapter parameters, enabling recovery of the original input context and facilitating safety. Empirical results on diverse multiple-choice and extractive question answering tasks show that CompAs outperforms ICL and prior generator-based methods, especially when scaling to more inputs. Our work establishes composable adapter generation as a practical and efficient alternative for scaling LLM deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
