通过逻辑引导合成评估LLM工具调用中的隐式监管合规性
Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis
AI总结：
本文提出LogiSafetyGen框架，通过逻辑引导合成评估LLM在工具调用中的隐式监管合规性，揭示大模型在安全约束上的不足。
摘要：
将大型语言模型（LLMs）整合到自主代理中已使复杂工具使用成为可能，但在高风险领域，这些系统必须严格遵守超出简单功能正确性的监管标准。然而，现有基准测试往往忽视隐式监管合规性，因此无法评估LLM是否能自主执行强制性安全约束。为填补这一空白，我们引入LogiSafetyGen框架，将无结构的监管转化为线性时间逻辑或acles，并采用逻辑引导的模糊测试来合成有效的、安全关键的轨迹。在此框架基础上，我们构建了LogiSafetyBench基准测试，包含240个人验证的任务，要求LLM生成满足功能目标和隐含合规规则的Python程序。对13种最先进的（SOTA）LLM进行评估发现，尽管较大模型在功能正确性上表现更好，但经常优先考虑任务完成而非安全，导致非合规行为。
英文摘要：
The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correctness. However, existing benchmarks often overlook implicit regulatory compliance, thus failing to evaluate whether LLMs can autonomously enforce mandatory safety constraints. To fill this gap, we introduce LogiSafetyGen, a framework that converts unstructured regulations into Linear Temporal Logic oracles and employs logic-guided fuzzing to synthesize valid, safety-critical traces. Building on this framework, we construct LogiSafetyBench, a benchmark comprising 240 human-verified tasks that require LLMs to generate Python programs that satisfy both functional objectives and latent compliance rules. Evaluations of 13 state-of-the-art (SOTA) LLMs reveal that larger models, despite achieving better functional correctness, frequently prioritize task completion over safety, which results in non-compliant behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
