As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focusing on reducing KV complexity and improving efficiency. For further research on expressiveness, we introduce our series of models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which aims to make RNN more expressive and demonstrates state tracking ability beyond transformers. We work with QRWK 32B based on RWKV-6 architecture, another approach that reduces the entire knowledge processing time to just 8 hours using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the distillation process can utilize any LLM, not just Qwen, and enables knowledge transfer from larger LLMs to smaller ones with more fewer tokens. We will explain the detailed process and share our insights on building more powerful foundation models. Please note that this is an ongoing work that will be updated continuously. The model checkpoints and source code are available at \href{https://github.com/yynil/RWKVInside}{https://github.com/yynil/RWKVInside}, \href{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
Linguistic Knowledge and Transferability of Contextual Representations
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Harvard John A. Paulson School of Engineering and Applied Sciences(哈佛大学约翰·A·保尔森工程与应用科学学院);MIT Computer Science and Artificial Intelligence Laboratory(麻省理工学院计算机科学与人工智能实验室);University of Washington(华盛顿大学)
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Microsoft(微软);DeepMind;Allen Institute for AI(艾伦人工智能研究所)
Pretraining Techniques for Sequence-to-Sequence Voice Conversion
Nagoya University(名古屋大学);Human Dataware Lab. Co. Ltd.(人类数据软件实验室有限公司);NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation(日本电信电话株式会社NTT通信科学实验室)
Native Hybrid Attention for Efficient Sequence Modeling
Tsinghua University(清华大学);Shanghai AI Laboratory(上海人工智能实验室);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Chinese University of Hong Kong(香港中文大学)
Gated Slot Attention for Efficient Linear-Time Sequence Modeling
School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Massachusetts Institute of Technology(麻省理工学院);University of California Santa Cruz(加州大学圣克鲁兹分校);Tencent AI Lab(腾讯AI实验室);LuxiTech;University of Waterloo(滑铁卢大学)