高效上下文扩展与LongCat ZigZag注意力
Efficient Context Scaling with LongCat ZigZag Attention
AI总结:
LongCat ZigZag Attention通过稀疏化全注意力模型,提升长上下文处理效率,实现高效长期推理和代理能力。
摘要:
我们介绍了LongCat ZigZag Attention(LoZA),这是一种稀疏注意力方案,旨在将任何现有的全注意力模型转换为稀疏版本,且计算预算有限。在长上下文场景中,LoZA可以在预填充密集(如检索增强生成)和解码密集(如工具集成推理)情况下实现显著加速。具体而言,通过在LongCat-Flash中期训练中应用LoZA,我们使LongCat-Flash-Exp成为一种长上下文基础模型,能够快速处理多达100万个token,从而实现高效的长期推理和长视界代理能力。
英文摘要:
We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。