arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.11564cs.CLcs.AI
收藏

上下文学科与性能相关性:分析在不同上下文长度下LLM性能及质量退化

Context Discipline and Performance Correlation: Analyzing LLM Performance and Quality Degradation Under Varying Context Lengths

Ahilan Ayyachamy Nadar Ponnusamy, Karthic Chandran, M Maruf Hossain

AI总结:

本文研究了不同上下文长度下LLM性能与质量退化的关系,发现KV缓存增长导致性能非线性退化,并揭示了MoE架构在高token体积时受基础设施瓶颈影响的问题。

摘要:

大型语言模型(LLM)的扩展趋势优先考虑增加最大上下文窗口,以促进复杂、长形式推理和文档分析。然而,管理扩展的上下文会引入严重的计算开销。本文研究了当密集型Transformer架构——特别是Llama-3.1-70B和Qwen1.5-14B——暴露于大量无关和分散的上下文时,系统性能与模型质量之间的关键权衡。研究发现,随着键值(KV)缓存的增长,性能退化呈现非线性特征。此外,对混合专家(MoE)架构的深入分析揭示了在不同上下文规模下出现的独特行为异常,表明在高token体积时,架构优势可能被基础设施瓶颈所掩盖。

英文摘要:

The scaling trend in Large Language Models (LLMs) has prioritized increasing the maximum context window to facilitate complex, long-form reasoning and document analysis. However, managing this expanded context introduces severe computational overhead. This paper investigates the critical trade-off between system performance and model quality when dense transformer architectures--specifically Llama-3.1-70B and Qwen1.5-14B--are exposed to large volumes of irrelevant and distracting context. The research identifies a non-linear performance degradation tied to the growth of the Key-Value (KV) cache. Furthermore, an extended analysis of the Mixture-of-Experts (MoE) architecture reveals unique behavioral anomalies at varying context scales, suggesting that architectural benefits may be masked by infrastructure bottlenecks at high token volumes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑