arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.15570cs.CL
收藏

ARWKV:预训练并非我们所求,一个源自 Transformer 的基于 RNN-注意力机制的语言模型

ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer

Lin Yueyu, Li Zhiyuan, Peter Yue, Liu Xiao

AI总结:

本文提出基于RWKV-7注意力机制的ARWKV模型及QRWK 32B蒸馏方法,利用16块AMD MI300X GPU在8小时内完成知识处理,在保持Qwen 2.5性能的同时增强了RNN的表达能力。

摘要:

众所周知,多头架构中的混合二次和次二次注意力模型已经超越了 Transformer 和 Linear RNN 模型,这些工作主要致力于降低 KV 复杂度并提升效率。为了进一步研究表达能力,我们介绍了从 Qwen 2.5 蒸馏出的一系列模型,这些模型基于纯原生的 RWKV-7 注意力机制,旨在使 RNN 具备更强的表达能力,并展现出超越 Transformer 的状态跟踪能力。我们研究了基于 RWKV-6 架构的 QRWK 32B,这是另一种方法,仅使用 16 块 AMD MI300X GPU 就将整个知识处理时间缩短至 8 小时,同时保持了 Qwen 2.5 的性能。事实上,蒸馏过程可以利用任何 LLM,而不仅仅是 Qwen,并且能够以更少的 token 实现从较大 LLM 向较小 LLM 的知识转移。我们将解释详细过程,并分享关于构建更强大基础模型的见解。请注意,这是一项正在进行的工作,将持续更新。模型检查点和源代码可在 https://github.com/yynil/RWKVInside 和 https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1 获取。

英文摘要:

As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focusing on reducing KV complexity and improving efficiency. For further research on expressiveness, we introduce our series of models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which aims to make RNN more expressive and demonstrates state tracking ability beyond transformers. We work with QRWK 32B based on RWKV-6 architecture, another approach that reduces the entire knowledge processing time to just 8 hours using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the distillation process can utilize any LLM, not just Qwen, and enables knowledge transfer from larger LLMs to smaller ones with more fewer tokens. We will explain the detailed process and share our insights on building more powerful foundation models. Please note that this is an ongoing work that will be updated continuously. The model checkpoints and source code are available at \href{https://github.com/yynil/RWKVInside}{https://github.com/yynil/RWKVInside}, \href{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑