arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.12689cs.NIcs.AIcs.ET
收藏

大语言模型能理解计算机网络吗?迈向虚拟系统管理员

Can LLMs Understand Computer Networks? Towards a Virtual System Administrator

University of Padova(帕多瓦大学);Spritz Matter Srl(斯普里茨马特有限公司);Delft University of Technology(代尔夫特理工大学)

Denis Donadel, Francesco Marchiori, Luca Pajola, Mauro Conti

AI总结:

本文首次系统评估了大语言模型理解计算机网络拓扑并回答相关问题的能力,通过构建评估框架在多种网络上测试专有与开源模型,发现最佳模型零样本平均准确率达79.3%,并指出复杂拓扑仍是主要挑战。

摘要:

人工智能,尤其是大语言模型(LLMs)的最新进展,为帮助系统管理员管理现代网络的复杂性提供了广阔前景。然而,尽管存在这种潜力,文献中关于LLMs能在多大程度上理解计算机网络仍存在显著空白。缺乏经验证据的情况下,系统管理员可能会依赖这些模型,却无法确信它们能准确执行网络相关任务。在本文中,我们首次对LLMs的计算机网络理解能力进行了详尽研究。我们提出了若干研究问题,以确定当提供网络拓扑及其相关问题时,LLMs能否给出正确答案。为评估这些问题,我们开发了一个完整的框架,用于评估LLMs在各种网络相关任务中的能力。我们在多个计算机网络上评估了该框架,采用了专有模型(如GPT4)和开源模型(如Llama2)。我们在零样本场景下对通用LLMs的研究结果显示出有希望的前景,最佳模型达到了79.3%的平均准确率。专有LLMs在小型和中型网络中取得了显著成果,但在理解复杂网络拓扑方面仍存在挑战,尤其是对开源模型而言。此外,我们还提供了关于提示工程如何提高某些任务准确率的见解。

英文摘要:

Recent advancements in Artificial Intelligence, and particularly Large Language Models (LLMs), offer promising prospects for aiding system administrators in managing the complexity of modern networks. However, despite this potential, a significant gap exists in the literature regarding the extent to which LLMs can understand computer networks. Without empirical evidence, system administrators might rely on these models without assurance of their efficacy in performing network-related tasks accurately. In this paper, we are the first to conduct an exhaustive study on LLMs' comprehension of computer networks. We formulate several research questions to determine whether LLMs can provide correct answers when supplied with a network topology and questions on it. To assess them, we developed a thorough framework for evaluating LLMs' capabilities in various network-related tasks. We evaluate our framework on multiple computer networks employing proprietary (e.g., GPT4) and open-source (e.g., Llama2) models. Our findings in general purpose LLMs using a zero-shot scenario demonstrate promising results, with the best model achieving an average accuracy of 79.3%. Proprietary LLMs achieve noteworthy results in small and medium networks, while challenges persist in comprehending complex network topologies, particularly for open-source models. Moreover, we provide insight into how prompt engineering can enhance the accuracy of some tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑