用于评估和加固LLM系统指令对抗编码攻击的自动化框架
Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks
AI总结：
本文提出自动化框架评估LLM系统指令在对抗编码攻击时的保密性，通过四个模型和46条指令测试发现结构化序列化攻击成功率高，提出基于Chain-of-Thought的缓解策略。
摘要：
大型语言模型（LLM）中的系统指令常用于执行安全策略、定义代理行为并保护敏感操作上下文。这些指令可能包含敏感信息如API凭证、内部政策和特权工作流定义，使系统指令泄露成为LLM应用中的关键安全风险。无需推理模型的开销，许多LLM应用依赖拒绝型指令来阻止直接请求系统指令，隐含假设被禁止的信息只能通过显式查询提取。我们引入了一个自动化评估框架，测试在将提取请求重新框架化为编码或结构化输出任务时系统指令是否保持保密。在四个常见模型和46条验证过的系统指令上，我们发现结构化序列化攻击的成功率（>0.7）。我们进一步展示了一种基于一次指令重塑的缓解策略，使用Chain-of-Thought推理模型，表明即使系统指令的措辞和结构有细微变化，也能显著降低攻击成功率，而无需重新训练模型。
英文摘要：
System Instructions in Large Language Models (LLMs) are commonly used to enforce safety policies, define agent behavior, and protect sensitive operational context in agentic AI applications. These instructions may contain sensitive information such as API credentials, internal policies, and privileged workflow definitions, making system instruction leakage a critical security risk highlighted in the OWASP Top 10 for LLM Applications. Without incurring the overhead costs of reasoning models, many LLM applications rely on refusal-based instructions that block direct requests for system instructions, implicitly assuming that prohibited information can only be extracted through explicit queries. We introduce an automated evaluation framework that tests whether system instructions remain confidential when extraction requests are re-framed as encoding or structured output tasks. Across seven common models and 46 verified system instructions, we observe high attack success rates (>0.7) for structured serialization where models refuse direct extraction requests but disclose protected content in the requested serialization formats. We further demonstrate a mitigation strategy based on one-shot instruction reshaping using a Chain-of-Thought reasoning model, indicating that even subtle changes in wording and structure of system instructions can significantly reduce attack success rate without requiring model retraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
