价值对齐 LLM 的意外危害：心理学与实证洞察
Unintended Harms of Value-Aligned LLMs: Psychological and Empirical Insights
AI总结：
本文实证分析价值对齐 LLM 的安全风险，结合心理学原理解释其按对齐价值生成文本而放大危害的机制，并提出上下文内对齐方法。
摘要：
大型语言模型（LLM）的应用范围持续扩大，使与人类价值对齐的个性化 LLM 受到越来越多关注。然而，使这些模型与个人价值对齐会引发重大安全担忧，因为某些价值可能与有害信息相关。本文识别了与价值对齐 LLM 相关的具体安全风险，并考察这些挑战背后的心理学原理。研究结果揭示两个关键洞见：（1）与未微调模型相比，价值对齐 LLM 更容易产生有害行为，并且在传统安全评估中表现出比其他微调模型略高的风险。（2）这些安全问题之所以出现，是因为价值对齐 LLM 确实会根据所对齐的价值生成文本，从而可能放大有害结果。利用一个包含详细安全类别的数据集，我们发现价值对齐与安全风险之间存在显著相关性，并有心理学假设提供支持。本研究为理解价值对齐的“黑箱”提供了洞见，并提出上下文内对齐方法以提升价值对齐 LLM 的安全性。
英文摘要：
The application scope of Large Language Models (LLMs) continues to expand, leading to increasing interest in personalized LLMs that align with human values. However, aligning these models with individual values raises significant safety concerns, as certain values may correlate with harmful information. In this paper, we identify specific safety risks associated with value-aligned LLMs and investigate the psychological principles behind these challenges. Our findings reveal two key insights. (1) Value-aligned LLMs are more prone to harmful behavior compared to non-fine-tuned models and exhibit slightly higher risks in traditional safety evaluations than other fine-tuned models. (2) These safety issues arise because value-aligned LLMs genuinely generate text according to the aligned values, which can amplify harmful outcomes. Using a dataset with detailed safety categories, we find significant correlations between value alignment and safety risks, supported by psychological hypotheses. This study offers insights into the "black box" of value alignment and proposes in-context alignment methods to enhance the safety of value-aligned LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
