停止测试攻击，开始诊断防御：四检查点框架揭示LLM安全为何破裂
Stop Testing Attacks, Start Diagnosing Defenses: The Four-Checkpoint Framework Reveals Where LLM Safety Breaks
AI总结：
本文提出四检查点框架，通过输入输出阶段和检测级别两个维度分析LLM安全机制，揭示防御漏洞所在及原因，提出WASR指标评估安全性能。
摘要：
大型语言模型（LLMs）部署安全机制以防止有害输出，但这些防御仍易受对抗性提示攻击。现有研究虽显示 jailbreak 攻击成功，但未解释防御为何失败及为何失败。为此，本文提出LLM安全作为连续流水线的假设，引入四检查点框架，沿处理阶段（输入 vs 输出）和检测级别（字面 vs 意图）两个维度组织安全机制，创建四个检查点（CP1-CP4），每个代表可独立评估的防御层。设计13种逃避技术，每种针对特定检查点，实现对个体防御层的可控测试。通过该框架，评估GPT-5、Claude Sonnet 4和Gemini 2.5 Pro在3312个单轮黑盒测试案例中的表现。采用LLM作为判断者的方法进行响应分类，并引入加权攻击成功率（WASR）指标，该指标是经过严重性调整的度量，捕捉二元评估所忽略的部分信息泄露。评估结果显示明显模式。传统二元ASR报告22.6%的攻击成功率。然而，WASR揭示52.7%，高出2.3倍。输出阶段防御（CP3、CP4）最弱，为72-79% WASR，而输入字面防御（CP1）最强，为13% WASR。Claude实现最强安全（42.8% WASR），随后是GPT-5（55.9%）和Gemini（59.5%）。这些发现表明，当前防御在输入字面检查点最强，但仍易受意图级操纵和输出阶段技术攻击。四检查点框架为识别和解决部署系统中的安全漏洞提供了结构化方法。
英文摘要：
Large Language Models (LLMs) deploy safety mechanisms to prevent harmful outputs, yet these defenses remain vulnerable to adversarial prompts. While existing research demonstrates that jailbreak attacks succeed, it does not explain \textit{where} defenses fail or \textit{why}. To address this gap, we propose that LLM safety operates as a sequential pipeline with distinct checkpoints. We introduce the \textbf{Four-Checkpoint Framework}, which organizes safety mechanisms along two dimensions: processing stage (input vs.\ output) and detection level (literal vs.\ intent). This creates four checkpoints, CP1 through CP4, each representing a defensive layer that can be independently evaluated. We design 13 evasion techniques, each targeting a specific checkpoint, enabling controlled testing of individual defensive layers. Using this framework, we evaluate GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro across 3,312 single-turn, black-box test cases. We employ an LLM-as-judge approach for response classification and introduce Weighted Attack Success Rate (WASR), a severity-adjusted metric that captures partial information leakage overlooked by binary evaluation. Our evaluation reveals clear patterns. Traditional Binary ASR reports 22.6\% attack success. However, WASR reveals 52.7\%, a 2.3$\times$ higher vulnerability. Output-stage defenses (CP3, CP4) prove weakest at 72--79\% WASR, while input-literal defenses (CP1) are strongest at 13\% WASR. Claude achieves the strongest safety (42.8\% WASR), followed by GPT-5 (55.9\%) and Gemini (59.5\%). These findings suggest that current defenses are strongest at input-literal checkpoints but remain vulnerable to intent-level manipulation and output-stage techniques. The Four-Checkpoint Framework provides a structured approach for identifying and addressing safety vulnerabilities in deployed systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
