arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.12196cs.CRcs.AI
收藏

利用大语言模型检测 npm 恶意软件包

Leveraging Large Language Models to Detect npm Malicious Packages

North Carolina State University(北卡罗来纳州立大学);Socket, Inc(Socket公司)

Nusrat Zahan, Philipp Burckhardt, Mikola Lysenko, Feross Aboukhadijeh, Laurie Williams

AI总结:

本文提出 SocketAI 恶意代码审查工作流,实证评估 GPT-3/GPT-4 检测 npm 恶意包,结合静态分析预筛选,显著提升精确率、F1 并降低分析成本。

摘要:

现有恶意代码检测技术需要集成多种工具以检测不同的恶意软件模式,且往往存在较高的误分类率。因此,可以通过采用先进的、自动化程度更高的方法来增强恶意代码检测技术,以实现高准确率和低误分类率。本研究的目标是通过实证研究大语言模型(LLM)在检测恶意代码方面的有效性,帮助安全分析师检测恶意软件包。我们提出了 SocketAI,这是一种用于检测恶意代码的恶意代码审查工作流。为评估 SocketAI 的有效性,我们使用了一个包含 5,115 个 npm 软件包的基准数据集,其中 2,180 个软件包含有恶意代码。我们使用先前研究中开发的 39 条自定义 CodeQL 规则来检测恶意 JavaScript 代码,将 GPT-3 和 GPT-4 模型与最先进的 CodeQL 静态分析工具进行了基线比较。我们还比较了将静态分析作为预筛选器与 SocketAI 工作流的有效性,衡量需要分析的文件数量以及相关成本。此外,我们开展了一项定性研究,以了解该工作流检测到或遗漏的恶意活动类型。基线比较表明,其精确率和 F1 分数分别比静态分析提高 16% 和 9%。GPT-4 实现了更高的准确率,精确率为 99%,F1 分数为 97%;而 GPT-3 以 91% 的精确率和 94% 的 F1 分数提供了更具成本效益的平衡。使用静态分析器预筛选文件,使需要 LLM 分析的文件数量减少 77.9%,并使 GPT-3 和 GPT-4 的成本分别降低 60.9% 和 76.1%。定性分析表明,数据窃取、任意代码执行和可疑域名类别是被检测出的主要恶意软件包类型。

英文摘要:

Existing malicious code detection techniques demand the integration of multiple tools to detect different malware patterns, often suffering from high misclassification rates. Therefore, malicious code detection techniques could be enhanced by adopting advanced, more automated approaches to achieve high accuracy and a low misclassification rate. The goal of this study is to aid security analysts in detecting malicious packages by empirically studying the effectiveness of Large Language Models (LLMs) in detecting malicious code. We present SocketAI, a malicious code review workflow to detect malicious code. To evaluate the effectiveness of SocketAI, we leverage a benchmark dataset of 5,115 npm packages, of which 2,180 packages have malicious code. We conducted a baseline comparison of GPT-3 and GPT-4 models with the state-of-the-art CodeQL static analysis tool, using 39 custom CodeQL rules developed in prior research to detect malicious Javascript code. We also compare the effectiveness of static analysis as a pre-screener with SocketAI workflow, measuring the number of files that need to be analyzed. and the associated costs. Additionally, we performed a qualitative study to understand the types of malicious activities detected or missed by our workflow. Our baseline comparison demonstrates a 16% and 9% improvement over static analysis in precision and F1 scores, respectively. GPT-4 achieves higher accuracy with 99% precision and 97% F1 scores, while GPT-3 offers a more cost-effective balance at 91% precision and 94% F1 scores. Pre-screening files with a static analyzer reduces the number of files requiring LLM analysis by 77.9% and decreases costs by 60.9% for GPT-3 and 76.1% for GPT-4. Our qualitative analysis identified data theft, execution of arbitrary code, and suspicious domain categories as the top detected malicious packages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑