arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.23966cs.CLcs.AI
收藏

高效上下文扩展与LongCat ZigZag注意力

Efficient Context Scaling with LongCat ZigZag Attention

Meituan, China(美团,中国)
Chen Zhang, Yang Bai, Jiahuan Li, Anchun Gui, Keheng Wang, Feifan Liu, Guanyu Wu, Yuwei Jiang, Defei Bu, Li Wei, Haihang Jing, Hongyin Tang, Xin Chen, Xiangzhou… 展开作者

Chen Zhang, Yang Bai, Jiahuan Li, Anchun Gui, Keheng Wang, Feifan Liu, Guanyu Wu, Yuwei Jiang, Defei Bu, Li Wei, Haihang Jing, Hongyin Tang, Xin Chen, Xiangzhou Huang, Fengcun Li, Rongxiang Weng, Yulei Qian, Yifan Lu, Yerui Sun, Jingang Wang, Yuchen Xie, Xunliang Cai

AI总结:

LongCat ZigZag Attention通过稀疏化全注意力模型,提升长上下文处理效率,实现高效长期推理和代理能力。

摘要:

我们介绍了LongCat ZigZag Attention(LoZA),这是一种稀疏注意力方案,旨在将任何现有的全注意力模型转换为稀疏版本,且计算预算有限。在长上下文场景中,LoZA可以在预填充密集(如检索增强生成)和解码密集(如工具集成推理)情况下实现显著加速。具体而言,通过在LongCat-Flash中期训练中应用LoZA,我们使LongCat-Flash-Exp成为一种长上下文基础模型,能够快速处理多达100万个token,从而实现高效的长期推理和长视界代理能力。

英文摘要:

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑