发表机构
eBay Inc.（eBay公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对ModelScan、ModelAudit和Fickling三类AI模型安全扫描器，在含170个人工制品的基准上评估其覆盖率与故障恢复能力，发现需区分判断准确性与可用性等指标。
AI 中文摘要
静态扫描器正越来越多地用于识别机器学习制品中可执行或其他不安全的内容，但传统评估指标仅能表征扫描器生成可用安全判断的情况。我们使用受人工制品支持的基准，在包含170个聚焦Pickle和PyTorch的人工制品（分为145个样本族）的合成语料库上，对ModelScan、ModelAudit和Fickling进行评估；其中135个样本族具有二进制安全真值，10个为无标签的故意畸形样本。我们明确区分非空值覆盖率、分析完成度、确定性安全决策、非安全发现及不支持的结果。在有标签的样本族中，ModelAudit对全部135个样本族生成了确定性安全决策（100%），Fickling为110个（81.5%），ModelScan为67个（49.6%）。在做出确定性判断的前提下，ModelScan的准确率、召回率和F1均达到100%；Fickling未识别出超出ModelAudit与ModelScan组合检测范围的独特真阳性样本族。此外，在ModelScan未能完成分析的48个恶意样本族中，ModelAudit和Fickling均生成了与真值一致的检测结果。这些发现凸显了将判断准确性与判断可用性、增量检测覆盖率与工具级冗余度分开的必要性。
英文摘要
Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, definitive security decisions, non-security findings, and unsupported outcomes. On labeled families, ModelAudit produced definitive security decisions for all 135 families (100%), Fickling for 110 (81.5%), and ModelScan for 67 (49.6%). Conditional on making a definitive judgment, ModelScan achieved 100% precision, recall, and F1. Fickling identified no unique true- positive families beyond those found by the combination of ModelAudit and ModelScan. Furthermore, for the 48 malicious families where ModelScan failed to complete its analysis, both ModelAudit and Fickling generated detections consistent with ground truth. These findings underscore the need to separate judgment accuracy from judgment availability, as well as incremental detection coverage from tool-level redundancy.
