Legend：利用表示工程为偏好数据集标注安全裕度
Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
AI总结：
本文提出Legend框架，利用表示工程在LLM嵌入空间构建安全方向，通过计算配对响应的语义距离自动标注偏好数据集的安全裕度，无需额外训练即可有效提升奖励建模和无害对齐效果。
摘要：
奖励模型在区分具有细微安全差异的响应方面的成功，关键取决于高质量的偏好数据集，该数据集应捕捉有害和无害响应的细粒度差异。这促使了开发包含偏好裕度的数据集的需求，该数据集应准确量化一个响应相对于另一个响应的无害程度。在本文中，我们迈出第一步，提出一个有效且具有成本效益的框架，以促进裕度增强的偏好数据集的发展。我们的框架Legend利用表示工程来标注偏好数据集。它在LLM的嵌入空间中构建代表安全的特定方向。通过利用这一安全方向，Legend随后可以利用配对响应沿此方向的语义距离来自动标注裕度。我们通过实验证明了Legend在奖励建模和LLM无害对齐方面的有效性。Legend还因其效率而脱颖而出，仅需推理时间而非额外训练。这种效率使得Legend更容易实现和扩展，使其在将LLM与安全对话对齐的实际应用中特别有价值。
英文摘要：
The success of the reward model in distinguishing between responses with subtle safety differences depends critically on the high-quality preference dataset, which should capture the fine-grained nuances of harmful and harmless responses. This motivates the need to develop a dataset involving preference margins, which accurately quantify how harmless one response is compared to another. In this paper, we take the first step to propose an effective and cost-efficient framework to promote the margin-enhanced preference dataset development. Our framework, Legend, Leverages representation engineering to annotate preference datasets. It constructs the specific direction within the LLM's embedding space that represents safety. By leveraging this safety direction, Legend can then leverage the semantic distances of paired responses along this direction to annotate margins automatically. We experimentally demonstrate our effectiveness in both reward modeling and harmless alignment for LLMs. Legend also stands out for its efficiency, requiring only the inference time rather than additional training. This efficiency allows for easier implementation and scalability, making Legend particularly valuable for practical applications in aligning LLMs with safe conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
