基于RSOD构建电信语言模型:领域数据、专家后训练、内存感知蒸馏及网络评估
Building Telecom Language Models with RSOD: Domain Data, Specialist Post-Training, Memory-Conscious Distillation, and Networking Evaluation
AI总结:
该研究提出RSOD框架构建电信语言模型,通过合成数据、LoRA教师训练及蒸馏整合,得到的Llama-CalixCLM v1在网络评估中性能接近多教师组合系统,具备部署可行性。
摘要:
电信语言模型需兼具广泛的网络知识与路由、宽带服务、接入技术及运营诊断等领域的专家行为,同时作为单一模型具备部署可行性。本文提出端到端框架,从结构化网络分类体系与专家图谱构建基于源的合成训练数据,生成14类网络路由下共212559条路由训练记录,在同一共享持续预训练主干上训练6个冻结的专家LoRA教师,通过路由专家在线蒸馏(RSOD)将其行为整合为可部署的学生适配器。在RSOD中,学生生成每条轨迹,路由分配的教师强制输出精确的原生token序列,通过共享冻结输出头重构紧凑的教师隐藏状态,仅更新学生LoRA。受控重构验证显示,原生与重构教师logits的平均绝对logit误差为7.26×10^-6,top-1一致性达100%。在4块RTX 6000 Ada GPU上,完成2000步运行生成16000条轨迹,未观测到学生版本或token ID不匹配。在2850条保留问题的路由条件评估中,原始Meta-Llama-3.1-8B-Instruct模型的微分数值为12.8661/25(宏分为13.0172/25),6个教师组合系统的微分为21.7641/25,整合后的RSOD学生(发布为Llama-CalixCLM v1)在同一5维度LLM评判标准下的微分为21.7112/25(宏分为21.8006/25);在类似部署的中立提示下,该学生的微分为20.0028/25(宏分为20.0778/25)。整合后的学生与6个教师组合系统的微分数值差不超过0.0529,同时保留经数值验证且可审计的训练-服务执行路径。
英文摘要:
Telecom language models must combine broad networking knowledge with specialist behavior across routing, broadband services, access technologies, and operational diagnostics while remaining practical to deploy as a single model. We present an end-to-end framework that builds source-grounded synthetic training data from structured networking taxonomies and expert graphs, materializes 212,559 routed training records across fourteen networking routes, trains six frozen specialist LoRA teachers over one shared continued-pretrained backbone, and consolidates their behavior into one deployable student adapter using Routed Specialist On-Policy Distillation (RSOD). In RSOD, the committed student generates each trajectory, the route-assigned specialist teacher-forces the exact native token sequence, compact teacher hidden states are reconstructed through the shared frozen output head, and only the student LoRA is updated. A controlled reconstruction validation shows mean absolute logit error of 7.26 x 10^-6 with 100% top-1 agreement between native and reconstructed teacher logits. On four RTX 6000 Ada GPUs, the completed 2,000-step run produced 16,000 trajectories with zero observed student-version or token- ID mismatches. Under the primary route-conditioned evaluation over 2,850 held-out questions, the raw Meta-Llama-3.1-8B-Instruct model scores 12.8661/25 micro (13.0172/25 macro), the combined routed six-teacher system scores 21.7641/25 micro, and the consolidated RSOD student, released as Llama-CalixCLM v1, reaches 21.7112/25 micro (21.8006/25 macro) on the same five dimension LLM-judge rubric. Under a separate neutral deployment-like prompt, the same student reaches 20.0028/25 micro (20.0778/25 macro). The consolidated student remains within 0.0529 micro-score points of the combined routed-teacher system while retaining a numerically validated and auditable train-serve execution path.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。