arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17358cs.CL
收藏

SoFA:基于优先规则遵循的即时屏蔽对齐

SoFA: Shielded On-the-fly Alignment via Priority Rule Following

University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);State Key Laboratory of Computer Science(计算机科学国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所)

Xinyu Lu, Bowen Yu, Yaojie Lu, Hongyu Lin, Haiyang Yu, Le Sun, Xianpei Han, Yongbin Li

AI总结:

本文提出优先规则遵循对齐范式及PriorityDistill方法,通过从LLM模拟中蒸馏优先遵循信号,使模型在对话中优先遵守规则而非用户指令,从而以极少的通用规则有效减少不对齐并抵御规则劫持。

摘要:

大语言模型(LLMs)中的对齐问题涉及使其适应广泛的人类价值观。由于偏好和监管标准的多样性,这一要求对现有对齐方法构成了挑战。本文提出了一种新的对齐范式——优先规则遵循,该范式将规则定义为每个对话中的主要控制机制,并使其优先于用户指令。我们的初步分析表明,即使是先进的LLMs(如GPT-4)在理解和优先处理规则方面也存在不足。因此,我们提出了PriorityDistill,这是一种半自动化方法,用于从LLM模拟中蒸馏优先遵循信号,以确保规则的稳健集成与遵守。实验表明,该方法不仅仅利用一条通用规则就能有效减少不对齐现象,还能平滑适应各种未见过的规则,确保这些规则免受劫持,并使模型做出适当响应。

英文摘要:

The alignment problem in Large Language Models (LLMs) involves adapting them to the broad spectrum of human values. This requirement challenges existing alignment methods due to diversity of preferences and regulatory standards. This paper introduces a novel alignment paradigm, priority rule following, which defines rules as the primary control mechanism in each dialog, prioritizing them over user instructions. Our preliminary analysis reveals that even the advanced LLMs, such as GPT-4, exhibit shortcomings in understanding and prioritizing the rules. Therefore, we present PriorityDistill, a semi-automated approach for distilling priority following signals from LLM simulations to ensure robust rule integration and adherence. Our experiments show that this method not only effectively minimizes misalignments utilizing only one general rule but also adapts smoothly to various unseen rules, ensuring they are shielded from hijacking and that the model responds appropriately.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑