可复现的多厂商DSM到CLI语义基准测试
A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation
AI总结:
针对网络意图翻译中语义正确性缺乏基准的问题,提出可复现的DSM-to-CLI语义基准,涵盖5种云LLM、3个厂商、5个用例,发现语义质量与可靠性正交,厂商效应主导,重复运行离散度预测投票不稳定。
摘要:
将高级网络意图转换为正确的多厂商配置仍然是网络自动化中的核心挑战,因为语法上有效的输出可能仍然违反预期的操作状态。尽管大型语言模型(LLM)最近取得了进展,但该领域仍然缺乏可复现的语义基准来进行严格的跨厂商评估。本文提出了一个可复现的DSM-to-CLI语义基准,涵盖五个云LLM、三个厂商、五个代表性用例,以及在固定评判者和明确失败分类下每个实验单元十次重复运行。我们的结果表明,语义质量和操作可靠性是正交的,厂商效应主导用例效应,重复运行离散度强烈预测投票不稳定性,其中华为VRP暴露了聚合指标隐藏的失败模式。这些发现表明,多厂商、重复执行的语义基准对于基于LLM的网络配置系统的科学严格比较至关重要。
英文摘要:
Translating high-level network intents into correct multivendor configurations remains a central challenge in network automation, as syntactically valid outputs may still violate the intended operational state. Despite recent advances in Large Language Models (LLMs), the field still lacks reproducible semantic benchmarks for rigorous cross-vendor evaluation. This paper presents a reproducible DSM-to-CLI semantic benchmark covering five cloud LLMs, three vendors, five representative use cases, and ten repeated runs per experimental cell under fixed judges and an explicit failure taxonomy. Our results show that semantic quality and operational reliability are orthogonal, vendor effects dominate use-case effects, and repeated-run dispersion strongly predicts vote instability, with Huawei VRP exposing failure modes hidden by aggregate metrics. These findings demonstrate that multivendor, repeated-execution semantic benchmarks are essential for scientifically rigorous comparison of LLM-based network configuration systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。