Mamba4Net:面向网络领域的蒸馏混合 Mamba 大语言模型
Mamba4Net: Distilled Hybrid Mamba Large Language Models For Networking
AI总结:
本文提出跨架构蒸馏框架 Mamba4Net,将 Transformer LLM 的网络知识迁移到线性复杂度 Mamba 学生模型,在三类网络任务中提升性能,并实现 3.96 倍吞吐量与仅 5.48% 的存储占用。
摘要:
基于 Transformer 的大语言模型(LLM)正越来越多地被用于网络研究,以应对特定领域挑战。然而,它们的二次时间复杂度和庞大的模型规模往往会造成显著的计算开销与内存限制,在资源受限环境中尤其如此。受知识蒸馏范式下 Deepseek-R1 模型的效率与性能启发,本文提出 Mamba4Net,这是一种新颖的跨架构蒸馏框架。Mamba4Net 将基于 Transformer 的 LLM 中的网络专用知识迁移到基于 Mamba 架构构建的学生模型,而 Mamba 架构具有线性时间复杂度。与基于 Transformer 的模型的二次复杂度相比,该设计显著提升了计算效率,同时缩小后的模型规模进一步降低了计算需求,改善了整体性能和资源利用率。为评估其有效性,Mamba4Net 在三类不同的网络任务上接受了测试:视口预测、自适应码率流媒体和集群作业调度。与不利用 LLM 的现有方法相比,Mamba4Net 展现出更优的任务性能。此外,相对于直接应用基于 Transformer 的 LLM,它实现了显著的效率提升,包括吞吐量提高至 3.96 倍,而存储占用仅为以往基于 LLM 方法所需存储的 5.48%。这些结果凸显了 Mamba4Net 在网络场景中实现高性价比应用 LLM 衍生知识的潜力。源代码已公开,以支持进一步的研究与开发。
英文摘要:
Transformer-based large language models (LLMs) are increasingly being adopted in networking research to address domain-specific challenges. However, their quadratic time complexity and substantial model sizes often result in significant computational overhead and memory constraints, particularly in resource-constrained environments. Drawing inspiration from the efficiency and performance of the Deepseek-R1 model within the knowledge distillation paradigm, this paper introduces Mamba4Net, a novel cross-architecture distillation framework. Mamba4Net transfers networking-specific knowledge from transformer-based LLMs to student models built on the Mamba architecture, which features linear time complexity. This design substantially enhances computational efficiency compared to the quadratic complexity of transformer-based models, while the reduced model size further minimizes computational demands, improving overall performance and resource utilization. To evaluate its effectiveness, Mamba4Net was tested across three diverse networking tasks: viewport prediction, adaptive bitrate streaming, and cluster job scheduling. Compared to existing methods that do not leverage LLMs, Mamba4Net demonstrates superior task performance. Furthermore, relative to direct applications of transformer-based LLMs, it achieves significant efficiency gains, including a throughput 3.96 times higher and a storage footprint of only 5.48% of that required by previous LLM-based approaches. These results highlight Mamba4Net's potential to enable the cost-effective application of LLM-derived knowledge in networking contexts. The source code is openly available to support further research and development.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。