不要命令，要培育：System-2对齐的探索性研究
Don't Command, Cultivate: An Exploratory Study of System-2 Alignment
AI总结：
本研究探究System-2思维模式对模型安全性的影响，评估o1模型的安全性能与漏洞，探索开源模型的System-2安全对齐方法，并提出过程监督的安全对齐实施方案。
摘要：
o1系统卡将o1模型列为OpenAI内部鲁棒性最强的模型，其核心特征是从快速直觉式思维向更缓慢、更审慎的推理演进。这一观察促使我们探究System-2思维模式对模型安全性的影响。在初步研究中，我们对o1模型开展了安全评估，涵盖使用对抗性自然语言提示词和数学编码提示词的复杂越狱攻击场景。研究结果表明，o1模型的安全性能有所提升，但仍存在漏洞，尤其在应对采用数学编码的越狱攻击时表现脆弱。通过详细的案例分析，我们识别出o1模型响应中的特定模式。我们还利用提示工程和监督微调技术，探索了开源模型中的System-2安全对齐问题。实验结果显示，一些鼓励模型仔细审查用户请求的简单方法对模型安全性有益。此外，我们提出了一项用于增强安全对齐的过程监督实施方案，具体实现细节和实验结果将在后续版本中公布。
英文摘要：
The o1 system card identifies the o1 models as the most robust within OpenAI, with their defining characteristic being the progression from rapid, intuitive thinking to slower, more deliberate reasoning. This observation motivated us to investigate the influence of System-2 thinking patterns on model safety. In our preliminary research, we conducted safety evaluations of the o1 model, including complex jailbreak attack scenarios using adversarial natural language prompts and mathematical encoding prompts. Our findings indicate that the o1 model demonstrates relatively improved safety performance; however, it still exhibits vulnerabilities, particularly against jailbreak attacks employing mathematical encoding. Through detailed case analysis, we identified specific patterns in the o1 model's responses. We also explored the alignment of System-2 safety in open-source models using prompt engineering and supervised fine-tuning techniques. Experimental results show that some simple methods to encourage the model to carefully scrutinize user requests are beneficial for model safety. Additionally, we proposed a implementation plan for process supervision to enhance safety alignment. The implementation details and experimental results will be provided in future versions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
