FalseReject：通过结构化推理提升上下文安全性并缓解LLMs过度拒绝的资源
FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
AI总结：
为缓解LLMs对良性查询的过度拒绝问题，本文提出FalseReject资源及图信息增强的对抗性多智能体框架，生成16k数据集以训练模型进行结构化推理，在保障整体安全性的同时大幅减少了不必要的拒绝。
摘要：
大型语言模型（LLMs）的安全对齐方法常导致对良性查询的过度拒绝，显著降低了其在敏感场景中的实用性。为应对此挑战，我们引入了FalseReject，这是一个包含16k看似有毒查询及跨44个安全相关类别结构化响应的综合资源。我们提出了一种图信息增强的对抗性多智能体交互框架来生成多样且复杂的提示，同时用显式推理构建响应，以帮助模型准确区分安全与不安全上下文。FalseReject包含为标准指令微调模型和推理导向模型量身定制的训练数据集，以及人工标注的基准测试集。我们对29个最先进（SOTA）的LLMs进行的广泛基准测试表明，过度拒绝挑战依然存在。实证结果表明，使用FalseReject进行监督微调能大幅减少不必要的拒绝，且不损害整体安全性或通用语言能力。
英文摘要：
Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
