从自动静态分析工具中检测误报：我们还有多远？
Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?
摘要：
自动静态分析工具（ASAT），如Findbugs，具有很高的误报率。产生的大量假警报对采用构成了障碍。研究人员已经提出使用机器学习来修剪错误警报，并仅向开发人员提供可操作的警告。这项最先进的研究已经确定了一组“黄金特性”，这些特性是基于对文件、代码和警告的特征和历史记录计算的度量标准。最近的研究表明，使用这些特征的机器学习非常有效，并且它们几乎达到了完美的性能。我们进行了详细的分析，以更好地了解“黄金特性”的强劲表现。我们发现，有几项研究使用了一种实验程序，导致数据泄漏和数据重复，这是具有重大意义的微妙问题。首先，地面事实标签已经泄露到测量给定上下文中可操作警告比例的功能中。其次，测试数据集中的许多警告出现在训练数据集中。接下来，我们展示了确定地面实况标签的警告预言中的局限性，这是一种将给定版本中的警告与未来的参考版本进行比较的启发式方法。我们发现，参考修订的选择影响的警告分布。此外，启发式产生的标签与人类的预言不一致。因此，以前看到的这些技术的强大性能是过于乐观的，如果在实践中采用它们的真实性能。我们的研究结果传达了几个教训，并提供评估虚警检测器的指导方针。
英文摘要：
Automatic static analysis tools (ASATs), such as Findbugs, have a high false alarm rate. The large number of false alarms produced poses a barrier to adoption. Researchers have proposed the use of machine learning to prune false alarms and present only actionable warnings to developers. The state-of-the-art study has identified a set of "Golden Features" based on metrics computed over the characteristics and history of the file, code, and warning. Recent studies show that machine learning using these features is extremely effective and that they achieve almost perfect performance. We perform a detailed analysis to better understand the strong performance of the "Golden Features". We found that several studies used an experimental procedure that results in data leakage and data duplication, which are subtle issues with significant implications. Firstly, the ground-truth labels have leaked into features that measure the proportion of actionable warnings in a given context. Secondly, many warnings in the testing dataset appear in the training dataset. Next, we demonstrate limitations in the warning oracle that determines the ground-truth labels, a heuristic comparing warnings in a given revision to a reference revision in the future. We show the choice of reference revision influences the warning distribution. Moreover, the heuristic produces labels that do not agree with human oracles. Hence, the strong performance of these techniques previously seen is overoptimistic of their true performance if adopted in practice. Our results convey several lessons and provide guidelines for evaluating false alarm detectors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
