LayerCake：大语言模型层内的Token感知对比解码
LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers
AI总结：
针对大语言模型易出现事实性错误的问题，提出token感知的层局部对比解码方法LayerCake，通过对齐token类型与对应Transformer层并选择性抑制注意力生成对比信号，无需训练即可在多模型多基准上提升生成事实性。
摘要：
大语言模型（LLMs）在自然语言理解与生成方面表现出色，但仍易出现事实性错误，限制了其在知识密集型任务中的可靠性。解码阶段策略无需训练即可提供颇具潜力的高效解决方案，但现有方法通常孤立处理token级和层级信号，忽略了二者之间的联合动态关系。本研究提出一种token感知的层局部对比解码方法，将特定token类型与其最具影响力的Transformer层对齐，以提升生成内容的事实性。通过实证注意力分析，我们发现两个关键模式：标点token在早期层获得主导注意力，而概念token在中间层主导语义推理。通过在对应深度选择性抑制对这些token类型的注意力，我们实现了可控的事实性退化诱导，并推导出对比信号以指导最终的事实性解码。本方法无需额外训练或模型修改，实验表明，该方法在多个LLM和各类基准上均能持续提升事实性。
英文摘要：
Large language models (LLMs) excel at natural language understanding and generation but remain vulnerable to factual errors, limiting their reliability in knowledge-intensive tasks. While decoding-time strategies provide a promising efficient solution without training, existing methods typically treat token-level and layer-level signals in isolation, overlooking the joint dynamics between them. In this work, we introduce a token-aware, layer-localized contrastive decoding method that aligns specific token types with their most influential transformer layers to improve factual generation. Through empirical attention analysis, we identify two key patterns: punctuation tokens receive dominant attention in early layers, while conceptual tokens govern semantic reasoning in intermediate layers. By selectively suppressing attention to these token types at their respective depths, we achieve the induction of controlled factual degradation and derive contrastive signals to guide the final factual decoding. Our method requires no additional training or model modification, and experiments demonstrate that our method consistently improves factuality across multiple LLMs and various benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
