arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.01255cs.CRcs.CLcs.SE
收藏

大语言模型无法可靠地检测JavaScript中的漏洞:首个系统性基准和评估

Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation

Lanzhou University(兰州大学);Zhejiang University(浙江大学);Beihang University(北京航空航天大学);Technology Support Center of the Cyberspace Administration of China(国家互联网信息办公室技术支撑中心);Fudan University(复旦大学);Beijing University of Posts(北京邮电大学)

Qingyuan Fei, Xin Liu, Song Li, Shujiang Wu, Jianwei Hou, Ping Chen, Zifeng Kang

AI总结:

本文首次提出FORGEJS框架,构建首个系统性JavaScript漏洞检测基准ARENAJS,并揭示LLM在漏洞检测中的局限性。

摘要:

研究人员已经提出了许多方法来检测JavaScript中的漏洞,尤其是那些由大语言模型(LLMs)辅助的方法。然而,LLMs在JavaScript漏洞检测中的实际能力仍存疑,需要系统性评估和全面基准。不幸的是,现有基准存在三个关键限制:(1)覆盖不全,例如只涵盖CWE类型的一个有限子集;(2)由于不合理的 ground truth 标签导致低估了LLM的能力;(3)由于不现实的案例,如使用孤立的漏洞文件而不是完整的项目而高估了能力。在本文中,我们首次引入了构建JavaScript漏洞检测基准的三个原则,直接解决这些限制:(1)全面性,(2)无低估,(3)无高估。基于这些原则,我们提出了FORGEJS,首个自动基准生成框架,用于评估LLM在JavaScript漏洞检测中的能力。然后,我们使用FORGEJS构建ARENAJS——首个系统性基准,用于评估基于LLM的JavaScript漏洞检测,并进一步提出JUDGEJS,一个自动评估框架。我们进行了首次系统性评估LLM在JavaScript漏洞检测中的能力,利用JUDGEJS在ARENAJS上评估了七种流行的商业LLM。结果表明,LLM不仅表现出有限的推理能力,还存在严重的鲁棒性缺陷,表明使用LLM进行可靠的JavaScript漏洞检测仍是一个开放性挑战。

英文摘要:

Researchers have proposed numerous methods to detect vulnerabilities in JavaScript, especially those assisted by Large Language Models (LLMs). However, the actual capability of LLMs in JavaScript vulnerability detection remains questionable, necessitating systematic evaluation and comprehensive benchmarks. Unfortunately, existing benchmarks suffer from three critical limitations: (1) incomplete coverage, such as covering a limited subset of CWE types; (2) underestimation of LLM capabilities caused by unreasonable ground truth labeling; and (3) overestimation due to unrealistic cases such as using isolated vulnerable files rather than complete projects. In this paper, we introduce, for the first time, three principles for constructing a benchmark for JavaScript vulnerability detection that directly address these limitations: (1) comprehensiveness, (2) no underestimation, and (3) no overestimation. Guided by these principles, we propose FORGEJS, the first automatic benchmark generation framework for evaluating LLMs' capability in JavaScript vulnerability detection. Then, we use FORGEJS to construct ARENAJS-the first systematic benchmark for LLM-based JavaScript vulnerability detection-and further propose JUDGEJS, an automatic evaluation framework. We conduct the first systematic evaluation of LLMs for JavaScript vulnerability detection, leveraging JUDGEJS to assess seven popular commercial LLMs on ARENAJS. The results show that LLMs not only exhibit limited reasoning capabilities, but also suffer from severe robustness defects, indicating that reliable JavaScript vulnerability detection with LLMs remains an open challenge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑