arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.15372cs.SEcs.AIcs.CR
收藏

SKILLS: 为基于LLM的电信运维进行结构化知识注入

SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

independent.academia.edu(独立学术教育)

Ivo Brett

AI总结:

研究探讨LLM在电信运维中执行API流程的可靠性,提出SKILLS框架,通过结构化知识指导提升模型性能,实验显示技能增强效果显著。

摘要:

随着电信运营商加速采用AI驱动的自动化,一个实际问题仍未解决:通用大语言模型(LLM)代理能否通过真实API接口可靠执行电信运维流程,还是需要结构化领域指导?我们介绍了SKILLS(为LLM驱动的服务生命周期操作进行结构化知识注入)基准框架,包含37个电信运维场景,覆盖8个TM Forum开放API领域(TMF620、TMF621、TMF622、TMF628、TMF629、TMF637、TMF639、TMF724)。每个场景均基于实时模拟API服务器,包含预置的生产代表性数据、MCP工具接口和确定性评估标准,结合响应内容检查、工具调用验证和数据库状态断言。我们评估了开放权重模型在两种条件下:基线(通用代理有工具访问但无领域指导)和带技能(代理增强带有可移植SKILL.md文档编码的流程逻辑、API模式和业务规则)。在5种开放权重模型条件和185次场景运行中,技能提升效果在所有模型中一致。MiniMax M2.5领先(81.1%带技能,+13.5个百分点),随后是Nemotron 120B(78.4%,+18.9个百分点)、GLM-5 Turbo(78.4%,+5.4个百分点)和Seed 2.0 Lite(75.7%,+18.9个百分点)

英文摘要:

As telecommunications operators accelerate adoption of AI-enabled automation, a practical question remains unresolved: can general-purpose large language model (LLM) agents reliably execute telecom operations workflows through real API interfaces, or do they require structured domain guidance? We introduce SKILLS (Structured Knowledge Injection for LLM-driven Service Lifecycle operations), a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains (TMF620, TMF621, TMF622, TMF628, TMF629, TMF637, TMF639, TMF724). Each scenario is grounded in live mock API servers with seeded production-representative data, MCP tool interfaces, and deterministic evaluation rubrics combining response content checks, tool-call verification, and database state assertions. We evaluate open-weight models under two conditions: baseline (generic agent with tool access but no domain guidance) and with-skill (agent augmented with a portable SKILL.md document encoding workflow logic, API patterns, and business rules). Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models. MiniMax M2.5 leads (81.1% with-skill, +13.5pp), followed by Nemotron 120B (78.4%, +18.9pp), GLM-5 Turbo (78.4%, +5.4pp), and Seed 2.0 Lite (75.7%, +18.9pp).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑