SecLens：针对安全漏洞检测LLM的角色特定评估
Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection
AI总结：
SecLens引入多利益相关者评估框架，通过角色特定加权配置评估LLM在安全漏洞检测中的性能差异，揭示多目标问题的本质。
摘要：
现有基于LLM的安全漏洞检测基准将模型性能压缩为单一指标，无法反映不同利益相关者的优先级差异。例如，CISO可能强调高召回率，工程主管可能优先减少误报，AI官可能平衡能力与成本。为解决这一限制，我们引入SecLens-R，一个围绕35个共享维度分组的7个测量类别的多利益相关者评估框架。该框架定义了五个角色特定的加权配置：CISO、首席AI官、安全研究员、工程主管和AI-as-Actor。每个配置选择12到16个维度，权重总和为80，产生一个0到100的复合决策分数。我们应用SecLens-R评估12个前沿模型，基于93个开源项目衍生的406个任务数据集，涵盖10种编程语言和8个OWASP对齐的漏洞类别。评估在Code-in-Prompt（CIP）和Tool-Use（TU）两种设置下进行。结果表明，不同利益相关者的视角下存在显著差异，相同模型的决策分数可能相差高达31分。例如，Qwen3-Coder在工程主管配置下得A（76.3），但在CISO配置下得D（45.2），而GPT-5.4也表现出类似差异。这些发现表明，漏洞检测本质上是多目标问题，利益相关者意识评估提供了单个聚合指标所掩盖的见解。
英文摘要：
Existing benchmarks for LLM-based vulnerability detection compress model performance into a single metric, which fails to reflect the distinct priorities of different stakeholders. For example, a CISO may emphasize high recall of critical vulnerabilities, an engineering leader may prioritize minimizing false positives, and an AI officer may balance capability against cost. To address this limitation, we introduce SecLens-R, a multi-stakeholder evaluation framework structured around 35 shared dimensions grouped into 7 measurement categories. The framework defines five role-specific weighting profiles: CISO, Chief AI Officer, Security Researcher, Head of Engineering, and AI-as-Actor. Each profile selects 12 to 16 dimensions with weights summing to 80, yielding a composite Decision Score between 0 and 100. We apply SecLens-R to evaluate 12 frontier models on a dataset of 406 tasks derived from 93 open-source projects, covering 10 programming languages and 8 OWASP-aligned vulnerability categories. Evaluations are conducted across two settings: Code-in-Prompt (CIP) and Tool-Use (TU). Results show substantial variation across stakeholder perspectives, with Decision Scores differing by as much as 31 points for the same model. For instance, Qwen3-Coder achieves an A (76.3) under the Head of Engineering profile but a D (45.2) under the CISO profile, while GPT-5.4 shows a similar disparity. These findings demonstrate that vulnerability detection is inherently a multi-objective problem and that stakeholder-aware evaluation provides insights that single aggregated metrics obscure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
