arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16304cs.CRcs.SE
收藏

注意差距:评估LLMs在高层面恶意包检测与细粒度指标识别中的表现

An Evaluation of Large Language Models for Detection of Malicious Python Packages

University of Alabama(阿拉巴马大学);University of Dhaka(达卡大学);Auburn High School(奥本高中);Auburn University(奥本大学)

Ahmed Ryan, Ibrahim Khalil, Abdullah Al Jahid, Md Erfan, Sungbin Park, Akond Ashfaque Ur Rahman, Md Rayhanur Rahman

AI总结:

本文评估了LLMs在恶意包检测与细粒度指标识别中的性能差异,发现通用模型在过滤威胁方面表现优异,但专用编码模型在识别严格结构的攻击时更有效。

摘要:

开源仓库中恶意包的普遍性,如PyPI,对软件供应链构成了重大威胁。尽管大型语言模型(LLMs)已出现作为自动化安全任务的有前途的工具,但它们在检测恶意包和指标方面的有效性仍处于探索阶段。本文系统评估了13个LLMs在检测恶意软件包方面的性能。使用一个精心编纂的4070个包的数据集(3700个良性和370个恶意),我们评估了模型在两个任务上的性能:二元分类(包检测)和多标签分类(特定恶意指标的识别)。我们进一步研究了提示策略、温度设置和模型规格对检测准确率的影响。我们发现LLMs能力存在显著的“粒度差距”。虽然GPT-4.1在二元检测中表现接近完美(F1≈0.99),当任务转移到识别特定恶意指标时,性能下降了约41%。我们观察到,通用模型在过滤大多数威胁方面表现最佳,而专门的编码模型在检测遵循严格、可预测的代码结构的攻击方面表现更好。我们的相关分析表明,参数大小和上下文宽度对检测准确率的解释力微乎其微。我们得出结论,尽管LLMs在包层面是强大的检测器,但它们缺乏识别细粒度指标所需的语义深度。

英文摘要:

Modern software development relies on open-source package repositories. Attackers use these to distribute malicious packages. Large Language Models (LLMs) can automatically detect these packages, but their ability to pinpoint specific malicious behaviors remains unclear. We evaluate 13 LLMs on two tasks using a dataset of 4,070 PyPI packages (370 malicious, 3,700 benign). The first task detects whether a package is malicious. The second identifies specific malicious indicators (lines of code). We evaluate each LLM across five prompt strategies and three temperatures. For the first task, LLMs achieve mean F1 scores from 0.40 to 0.99, detecting most malicious packages but frequently flagging safe ones. For the second task, LLMs achieve a weighted F1 score of 0.69 for recognizing behavior types, dropping to 0.48 when identifying specific indicators. LLMs recognize standard code patterns but miss indicators requiring broader context or the author's intent. LLMs also report absent indicators. Evaluating the association between performance and model size, context width, prompt strategy, temperature, and code complexity reveals only code complexity has a meaningful impact: longer packages are harder to analyze. We recommend using LLMs for initial triage to flag suspicious packages for human review, rather than identifying specific malicious mechanisms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑