揭示初始显著性在U形注意力偏差中的作用：通过缩放初始标记权重提升长文本处理
Uncovering the Role of Initial Saliency in U-Shaped Attention Bias: Scaling Initial Token Weight for Enhanced Long-Text Processing
AI总结：
本文提出通过缩放初始标记权重以提升长文本处理能力，发现初始显著性对U形注意力偏差有重要影响，并在多个任务中取得性能提升。
摘要：
大型语言模型（LLMs）在多种自然语言处理（NLP）任务上表现优异。然而，它们在处理长文本序列时常常受到
英文摘要：
Large language models (LLMs) have demonstrated strong performance on a variety of natural language processing (NLP) tasks. However, they often struggle with long-text sequences due to the ``lost in the middle'' phenomenon. This issue has been shown to arise from a U-shaped attention bias, where attention is disproportionately focused on the beginning and end of a text, leaving the middle section underrepresented. While previous studies have attributed this bias to position encoding, our research first identifies an additional factor: initial saliency. It means that in the attention computation for each token, tokens with higher attention weights relative to the initial token tend to receive more attention in the prediction of the next token. We further find that utilizing this property by scaling attention weight between the initial token and others improves the model's ability to process long contexts, achieving a maximum improvement of 3.6\% in MDQA dataset. Moreover, combining this approach with existing methods to reduce position encoding bias further enhances performance, achieving a maximum improvement of 3.4\% in KV-Retrieval tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
