Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation
Lanzhou University(兰州大学);Zhejiang University(浙江大学);Beihang University(北京航空航天大学);Technology Support Center of the Cyberspace Administration of China(国家互联网信息办公室技术支撑中心);Fudan University(复旦大学);Beijing University of Posts(北京邮电大学)
Qingyuan Fei, Xin Liu, Song Li, Shujiang Wu, Jianwei Hou, Ping Chen, Zifeng Kang
研究人员已经提出了许多方法来检测JavaScript中的漏洞,尤其是那些由大语言模型(LLMs)辅助的方法。然而,LLMs在JavaScript漏洞检测中的实际能力仍存疑,需要系统性评估和全面基准。不幸的是,现有基准存在三个关键限制:(1)覆盖不全,例如只涵盖CWE类型的一个有限子集;(2)由于不合理的 ground truth 标签导致低估了LLM的能力;(3)由于不现实的案例,如使用孤立的漏洞文件而不是完整的项目而高估了能力。在本文中,我们首次引入了构建JavaScript漏洞检测基准的三个原则,直接解决这些限制:(1)全面性,(2)无低估,(3)无高估。基于这些原则,我们提出了FORGEJS,首个自动基准生成框架,用于评估LLM在JavaScript漏洞检测中的能力。然后,我们使用FORGEJS构建ARENAJS——首个系统性基准,用于评估基于LLM的JavaScript漏洞检测,并进一步提出JUDGEJS,一个自动评估框架。我们进行了首次系统性评估LLM在JavaScript漏洞检测中的能力,利用JUDGEJS在ARENAJS上评估了七种流行的商业LLM。结果表明,LLM不仅表现出有限的推理能力,还存在严重的鲁棒性缺陷,表明使用LLM进行可靠的JavaScript漏洞检测仍是一个开放性挑战。
英文摘要:
Researchers have proposed numerous methods to detect vulnerabilities in JavaScript, especially those assisted by Large Language Models (LLMs). However, the actual capability of LLMs in JavaScript vulnerability detection remains questionable, necessitating systematic evaluation and comprehensive benchmarks. Unfortunately, existing benchmarks suffer from three critical limitations: (1) incomplete coverage, such as covering a limited subset of CWE types; (2) underestimation of LLM capabilities caused by unreasonable ground truth labeling; and (3) overestimation due to unrealistic cases such as using isolated vulnerable files rather than complete projects.
In this paper, we introduce, for the first time, three principles for constructing a benchmark for JavaScript vulnerability detection that directly address these limitations: (1) comprehensiveness, (2) no underestimation, and (3) no overestimation. Guided by these principles, we propose FORGEJS, the first automatic benchmark generation framework for evaluating LLMs' capability in JavaScript vulnerability detection. Then, we use FORGEJS to construct ARENAJS-the first systematic benchmark for LLM-based JavaScript vulnerability detection-and further propose JUDGEJS, an automatic evaluation framework.
We conduct the first systematic evaluation of LLMs for JavaScript vulnerability detection, leveraging JUDGEJS to assess seven popular commercial LLMs on ARENAJS. The results show that LLMs not only exhibit limited reasoning capabilities, but also suffer from severe robustness defects, indicating that reliable JavaScript vulnerability detection with LLMs remains an open challenge.
Obfuscating Code Vulnerabilities against Static Analysis in JavaScript Code
University of Verona(维罗纳大学);University of Cagliari(卡利亚里大学);School of Advanced Defense Studies (CASD)(高级防务研究学院(CASD));National Interuniversity Consortium for Informatics(国家大学间信息学联盟)
A Simple and Efficient Jailbreak Method Exploiting LLMs' Helpfulness
The Harbin Institute of Technology(哈尔滨工业大学);The Hong Kong Polytechnic University(香港理工大学);Shenzhen University(深圳大学);Shenzhen Loop Area Institute(深圳河套学院)
Detile: Fine-Grained Information Leak Detection in Script Engines
内存泄漏攻击在利用内存损坏漏洞方面发挥着重要作用。通过分析最近的研究,我们观察到,绕过强制控制流完整性或尝试检测面向返回的编程的防御性解决方案需要将内存泄露攻击作为基本的第一步。然而,在检测此类信息泄漏方面的研究滞后。在本文中,我们解决了这个问题,并提出了一个细粒度的,自动检测的内存泄漏攻击脚本引擎的系统。基本观点如下:脚本语言,如Web浏览器中的JavaScript,是严格沙箱化的。他们必须不提供任何关于其上下文中的内存布局的见解。事实上,任何此类信息都可能代表正在进行的内存泄露攻击。因此,为了检测信息泄漏,我们的系统创建了一个克隆的脚本引擎进程与重新随机化的内存布局。克隆被检测为与原始进程同步。当执行内存泄漏以泄漏有关内存布局的信息时,两个进程的脚本上下文中会出现任何不一致。基于这种检测方法,我们已经设计并实现了Detile(\underline{i}信息\underline{le} aks的\underline{det}部分),在Windows 8.0/8.1上的Microsoft Internet Explorer 10/11中的JavaScript引擎的原型。实证评估表明,我们的工具可以成功地检测内存泄漏攻击,即使对这个专有软件。
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
Sun Yat-sen University(中山大学);Nanyang Technological University(南洋理工大学);Alibaba Group(阿里巴巴集团);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室)