Attn-GS：基于注意力的上下文压缩用于高效个性化大语言模型
Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs
AI总结：
Attn-GS通过利用LLMs的注意力模式实现高效个性化，通过标记模型标记重要句子并指导压缩模型生成高质量压缩上下文，显著提升性能并减少token使用。
摘要：
对大型语言模型（LLMs）进行个性化需要整合大量的交互历史和资料，但由于输入token限制，这在实践中变得不切实际，因为推理延迟和API成本过高。现有方法依赖于启发式方法，如选择最近的交互或提示摘要模型来压缩用户资料。然而，这些方法将上下文视为整体，未能考虑LLMs如何内部处理和优先不同资料组件。我们研究LLMs的注意力模式是否能有效识别重要的个性化信号以实现智能上下文压缩。通过在代表性个性化任务上的初步研究，我们发现（a）LLMs的注意力模式自然揭示了重要的信号，（b）微调增强了LLMs区分相关和不相关信息的能力。基于这些见解，我们提出了Attn-GS，一个基于注意力的上下文压缩框架，利用标记模型的注意力反馈来标记重要的个性化句子，然后指导压缩模型生成任务相关的高质量压缩用户上下文。大量实验表明，Attn-GS在不同任务、token限制和设置中显著优于各种基线，实现了接近使用完整上下文的性能，同时将token使用量减少50倍。
英文摘要：
Personalizing large language models (LLMs) to individual users requires incorporating extensive interaction histories and profiles, but input token constraints make this impractical due to high inference latency and API costs. Existing approaches rely on heuristic methods such as selecting recent interactions or prompting summarization models to compress user profiles. However, these methods treat context as a monolithic whole and fail to consider how LLMs internally process and prioritize different profile components. We investigate whether LLMs' attention patterns can effectively identify important personalization signals for intelligent context compression. Through preliminary studies on representative personalization tasks, we discover that (a) LLMs' attention patterns naturally reveal important signals, and (b) fine-tuning enhances LLMs' ability to distinguish between relevant and irrelevant information. Based on these insights, we propose Attn-GS, an attention-guided context compression framework that leverages attention feedback from a marking model to mark important personalization sentences, then guides a compression model to generate task-relevant, high-quality compressed user contexts. Extensive experiments demonstrate that Attn-GS significantly outperforms various baselines across different tasks, token limits, and settings, achieving performance close to using full context while reducing token usage by 50 times.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
