PKU-SafeRLHF：面向大语言模型的人类偏好多级安全对齐研究
PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
AI总结：
本研究推出大规模安全人类偏好数据集PKU-SafeRLHF，分离有用性与无害性标注，覆盖多危害类别与严重等级，并基于该数据训练安全对齐相关模型，为LLM安全部署提供资源支撑。
摘要：
本研究推出安全人类偏好数据集PKU-SafeRLHF，旨在推动大语言模型（LLM）安全对齐领域的研究。作为SafeRLHF与BeaverTails的姊妹项目，我们将问答对的有用性与无害性标注分离，为这两个耦合属性提供差异化的研究视角。整体而言，我们提供了4.46万条经过精炼的提示词、26.5万条带有安全元标签的问答对，覆盖19个危害类别以及从轻微到严重的三个严重程度等级，所有回答均由Llama系列模型生成。在此基础上，我们收集了16.68万条偏好数据，分别包含双偏好数据（有用性与无害性解耦）和单偏好数据（从零开始权衡有用性与无害性）。依托该大规模标注数据，我们进一步训练了用于LLM风险控制的严重程度敏感审核模型，以及用于LLM安全对齐的以安全为核心的RLHF算法。我们认为该数据集将成为社区的宝贵资源，助力LLM的安全部署。数据可在https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF获取。
英文摘要：
In this study, we introduce the safety human preference dataset, PKU-SafeRLHF, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs. Data is available at https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
