arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.13846cs.CLcs.AIcs.LG
收藏

LightTransfer: 你的长上下文LLM实际上是一个具有轻松适应能力的混合模型

LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation

Singapore Management University(新加坡国立大学);National University of Singapore(新加坡国立大学);Sea AI Lab, Singapore(新加坡海智实验室)

Xuan Zhang, Fengzhuo Zhang, Cunxiao Du, Chao Du, Tianyu Pang, Wei Gao, Min Lin

AI总结:

本文提出LightTransfer方法,通过将LLaMA等模型转换为混合架构,实现更高效的生成,实验表明在长上下文理解任务中,即使有半数层被识别为懒层,也能在性能损失小于1.5%的情况下提升2.17倍的吞吐量,并在数学基准AIME24上达到53.3%的分数。

摘要:

将语言模型扩展到处理更长上下文引入了由于键值(KV)缓存成本增加而带来的显著内存挑战。受混合模型的效率提升和预训练大变压器骨干的广泛可用性启发,我们探索将变压器模型转换为混合架构以实现更高效的生成。在本工作中,我们提出了LightTransfer,一种轻量级方法,将模型如LLaMA转换为混合变体。我们的方法识别出懒层——那些专注于最近或初始token的层,并将它们的完整注意力替换为流式注意力。这种转换可以在无需任何训练的情况下用于长上下文理解任务,或在需要更强推理能力的o1-like长推理生成任务中进行最小微调。在多样化的基准和模型(如LLaMA、Mistral、QwQ-STILL)上的实验表明,即使有半数层被识别为懒层,LightTransfer在性能损失小于1.5%(在LongBench上)的情况下,也能实现高达2.17倍的吞吐量提升,并在数学基准AIME24上达到先进o1-like长推理模型QwQ-STILL的53.3%。

英文摘要:

Scaling language models to handle longer contexts introduces substantial memory challenges due to the growing cost of key-value (KV) caches. Motivated by the efficiency gains of hybrid models and the broad availability of pretrained large transformer backbones, we explore transitioning transformer models into hybrid architectures for a more efficient generation. In this work, we propose LightTransfer, a lightweight method that transforms models such as LLaMA into hybrid variants. Our approach identifies lazy layers -- those focusing on recent or initial tokens -- and replaces their full attention with streaming attention. This transformation can be performed without any training for long-context understanding tasks or with minimal fine-tuning for o1-like long reasoning generation tasks that require stronger reasoning capabilities. Experiments across diverse benchmarks and models (e.g., LLaMA, Mistral, QwQ-STILL) demonstrate that, even when half of the layers are identified as lazy, LightTransfer achieves up to 2.17$\times$ throughput improvement with minimal performance loss ($<1.5\%$ on LongBench) and achieves 53.3\% on math benchmark AIME24 of advanced o1-like long reasoning model QwQ-STILL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑