Modern software development relies on open-source package repositories. Attackers use these to distribute malicious packages. Large Language Models (LLMs) can automatically detect these packages, but their ability to pinpoint specific malicious behaviors remains unclear. We evaluate 13 LLMs on two tasks using a dataset of 4,070 PyPI packages (370 malicious, 3,700 benign). The first task detects whether a package is malicious. The second identifies specific malicious indicators (lines of code). We evaluate each LLM across five prompt strategies and three temperatures.
For the first task, LLMs achieve mean F1 scores from 0.40 to 0.99, detecting most malicious packages but frequently flagging safe ones. For the second task, LLMs achieve a weighted F1 score of 0.69 for recognizing behavior types, dropping to 0.48 when identifying specific indicators. LLMs recognize standard code patterns but miss indicators requiring broader context or the author's intent. LLMs also report absent indicators. Evaluating the association between performance and model size, context width, prompt strategy, temperature, and code complexity reveals only code complexity has a meaningful impact: longer packages are harder to analyze. We recommend using LLMs for initial triage to flag suspicious packages for human review, rather than identifying specific malicious mechanisms.
Cross-Ecosystem Vulnerability Analysis for Python Applications
University of Athens(雅典大学);National Infrastructures for Research and Technology(国家研究与技术基础设施);Athens University of Economics and Business(雅典经济与商业大学);ETH Zurich(苏黎世联邦理工学院);Technical University of Crete(克里特理工大学)
DistillGuard: Malicious NPM Package Detection and API Attack Chain Analysis via Static Graph and LLM Distillation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
PYPILINE: Malicious PyPI Package Detection via Suspicious API Knowledge and Agent Workflow
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)