OR-Bench：面向大语言模型的过度拒绝基准
OR-Bench: An Over-Refusal Benchmark for Large Language Models
AI总结：
针对大语言模型安全对齐后易过度拒绝无害提示的问题，研究提出自动生成大规模过度拒绝数据集的方法，构建了含8万余条提示的OR-Bench基准，并测评了32款主流大模型，相关资源已开源以助力安全对齐研究。
摘要：
大语言模型（LLMs）需要经过严谨的安全对齐以避免生成恶意输出。尽管大量研究聚焦于减少有害内容生成，但安全性能的提升往往伴随过度拒绝的副作用——LLMs可能会拒绝无害的提示，实用性有所下降。虽然过度拒绝问题已在实验中被观测到，但由于难以构造能诱发LLMs过度拒绝行为的提示，对其进行系统性测量颇具挑战。本研究提出了一种自动生成大规模过度拒绝数据集的新方法。借助该技术，我们推出了首个大规模过度拒绝基准OR-Bench。OR-Bench包含覆盖10个常见拒答类别的80000条过度拒绝提示、约1000条即便对前沿LLMs也颇具难度的困难提示子集，以及额外600条有毒提示以避免模型无差别响应。随后我们开展了综合研究，测量了8个模型家族共32款主流LLMs的过度拒绝情况。我们的数据集公开发布于https://huggingface.co/bench-llms，代码库开源于https://github.com/justincui03/or-bench。我们希望该基准能助力社区开发出更优质的安全对齐模型。
英文摘要：
Large Language Models (LLMs) require careful safety alignment to prevent malicious outputs. While significant research focuses on mitigating harmful content generation, the enhanced safety often come with the side effect of over-refusal, where LLMs may reject innocuous prompts and become less helpful. Although the issue of over-refusal has been empirically observed, a systematic measurement is challenging due to the difficulty of crafting prompts that can elicit the over-refusal behaviors of LLMs. This study proposes a novel method for automatically generating large-scale over-refusal datasets. Leveraging this technique, we introduce OR-Bench, the first large-scale over-refusal benchmark. OR-Bench comprises 80,000 over-refusal prompts across 10 common rejection categories, a subset of around 1,000 hard prompts that are challenging even for state-of-the-art LLMs, and an additional 600 toxic prompts to prevent indiscriminate responses. We then conduct a comprehensive study to measure the over-refusal of 32 popular LLMs across 8 model families. Our datasets are publicly available at https://huggingface.co/bench-llms and our codebase is open-sourced at https://github.com/justincui03/or-bench. We hope this benchmark can help the community develop better safety aligned models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
