arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16100cs.CRcs.DC
收藏

LLMSecCode:评估用于安全编码的大型语言模型

LLMSecCode: Evaluating Large Language Models for Secure Coding

Anton Rydén, Erik Näslund, Elad Michael Schiller, Magnus Almgren

AI总结:

本文提出开源评估框架LLMSecCode,旨在客观评估大型语言模型在安全编码方面的能力,通过实验验证了参数和提示对性能的影响,并鼓励LLM安全能力的标准化与基准测试。

摘要:

大型语言模型(LLMs)的快速部署需要仔细考虑其对网络安全的影响。我们的工作旨在改进适合促进安全编码(SC)的LLM选择过程。这引发了具有挑战性的研究问题,例如(RQ1)哪些功能可以简化LLM评估?(RQ2)评估应该衡量什么?(RQ3)如何证明评估过程是公正的?为了解决这些问题,我们引入了LLMSecCode,这是一个旨在客观评估LLM SC能力的开源评估框架。我们通过实验验证了LLMSecCode的实现。在改变参数和提示时,我们发现性能分别存在10%和9%的差异。我们还将部分结果与可靠的外部参与者进行了比较,结果显示存在5%的差异。我们努力确保开源框架的易用性,并鼓励外部参与者进一步开发。借助LLMSecCode,我们希望鼓励对LLM在面向安全的代码和任务中的能力进行标准化和基准测试。

英文摘要:

The rapid deployment of Large Language Models (LLMs) requires careful consideration of their effect on cybersecurity. Our work aims to improve the selection process of LLMs that are suitable for facilitating Secure Coding (SC). This raises challenging research questions, such as (RQ1) Which functionality can streamline the LLM evaluation? (RQ2) What should the evaluation measure? (RQ3) How to attest that the evaluation process is impartial? To address these questions, we introduce LLMSecCode, an open-source evaluation framework designed to assess LLM SC capabilities objectively. We validate the LLMSecCode implementation through experiments. When varying parameters and prompts, we find a 10% and 9% difference in performance, respectively. We also compare some results to reliable external actors, where our results show a 5% difference. We strive to ensure the ease of use of our open-source framework and encourage further development by external actors. With LLMSecCode, we hope to encourage the standardization and benchmarking of LLMs' capabilities in security-oriented code and tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑