谁来评估评估者?关于评估基于人工智能的进攻性代码生成器的自动度量标准研究
Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators
AI总结:
本文系统分析了多种输出相似性自动度量标准在进攻性代码生成器评估中的适用性,通过在两个NMT模型和两个数据集上的实验与人工评估对比,揭示了各度量标准的优势与局限。
摘要:
基于人工智能的代码生成器是一种新兴的解决方案,它通过使用深度神经网络(神经机器翻译,NMT)从自然语言描述开始自动编写程序。特别是,代码生成器已被用于通过生成概念验证攻击来进行道德黑客攻击和进攻性安全测试。不幸的是,代码生成器的评估仍然面临若干问题。当前的做法使用输出相似性度量,即计算生成代码与真实参考之间文本相似性的自动度量标准。然而,目前尚不清楚应该使用哪种度量标准,以及哪种度量标准最适合特定场景。本工作分析了大量针对进攻性代码生成器的输出相似性度量标准。我们将这些度量标准应用于两个最先进的NMT模型,使用两个包含进攻性汇编代码和Python代码及其英文描述的数据集。我们将自动度量标准的估计结果与人工评估进行比较,并对其优势和局限性提供了实用的见解。
英文摘要:
AI-based code generators are an emerging solution for automatically writing programs starting from descriptions in natural language, by using deep neural networks (Neural Machine Translation, NMT). In particular, code generators have been used for ethical hacking and offensive security testing by generating proof-of-concept attacks. Unfortunately, the evaluation of code generators still faces several issues. The current practice uses output similarity metrics, i.e., automatic metrics that compute the textual similarity of generated code with ground-truth references. However, it is not clear what metric to use, and which metric is most suitable for specific contexts. This work analyzes a large set of output similarity metrics on offensive code generators. We apply the metrics on two state-of-the-art NMT models using two datasets containing offensive assembly and Python code with their descriptions in the English language. We compare the estimates from the automatic metrics with human evaluation and provide practical insights into their strengths and limitations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。