arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.04602cs.SEcs.AIcs.IR
收藏

代码遵从性评估是一个学习问题

Code Compliance Assessment as a Learning Problem

Amazon(亚马逊)

Neela Sawant, Srinivasan H. Sengamedu

摘要:

手动代码审查和静态代码分析器是验证源代码是否符合编码策略的传统机制。然而,这些机制很难扩展。我们将代码合规性评估制定为机器学习(ML)问题,以自然语言策略和代码作为输入,并生成对代码的合规性、不合规性或不相关性的预测。这可以帮助扩展合规性分类并搜索传统机制未涵盖的策略。我们探讨了ML模型制定,训练数据和评估设置的关键研究问题。其核心思想是通过代码和策略嵌入的向量距离来获得一个联合的代码-文本嵌入空间,该空间保持了遵从性关系。由于没有特定于任务的数据,我们重新解释和过滤常用的软件数据集,并增加额外的预训练和预微调任务,以减少语义差距。我们对两个编码政策列表(CWE和CBP)进行了基准测试。这是一个zero-shot评估,因为在训练集中没有策略出现。在CWE和CBP上,我们的工具Policy 2Code分别实现了分类准确率(59%,71%)和搜索MRR(0.05,0.21),而CodeBERT的分类准确率为(37%,54%)和MRR(0.02,0.02)。在一项用户研究中,24%的Policy 2Code检测被接受,而CodeBERT只有7%。

英文摘要:

Manual code reviews and static code analyzers are the traditional mechanisms to verify if source code complies with coding policies. However, these mechanisms are hard to scale. We formulate code compliance assessment as a machine learning (ML) problem, to take as input a natural language policy and code, and generate a prediction on the code's compliance, non-compliance, or irrelevance. This can help scale compliance classification and search for policies not covered by traditional mechanisms. We explore key research questions on ML model formulation, training data, and evaluation setup. The core idea is to obtain a joint code-text embedding space which preserves compliance relationships via the vector distance of code and policy embeddings. As there is no task-specific data, we re-interpret and filter commonly available software datasets with additional pre-training and pre-finetuning tasks that reduce the semantic gap. We benchmarked our approach on two listings of coding policies (CWE and CBP). This is a zero-shot evaluation as none of the policies occur in the training set. On CWE and CBP respectively, our tool Policy2Code achieves classification accuracies of (59%, 71%) and search MRR of (0.05, 0.21) compared to CodeBERT with classification accuracies of (37%, 54%) and MRR of (0.02, 0.02). In a user study, 24% Policy2Code detections were accepted compared to 7% for CodeBERT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑