DeAL：大型语言模型的解码时间对齐
DeAL: Decoding-time Alignment for Large Language Models
AI总结：
DeAL通过允许用户自定义奖励函数和实现解码时间对齐，改进了大型语言模型对齐目标的实现。
摘要：
大型语言模型（LLMs）如今被期望生成符合人类偏好的内容。当前的研究主要集中在训练时间的对齐上，通过诸如人类反馈强化学习（RLHF）等技术。然而，这种技术是否是教授模型对齐目标的有效方法仍不清楚。首先，无法整合多种自定义奖励以及依赖模型开发者对普遍和静态原则的看法是关键限制。其次，此类方法的可靠性也值得怀疑（例如，即使经过安全训练，也容易受到劫持攻击）。为了解决这些问题，我们提出了DeAL框架，允许用户自定义奖励函数，并实现LLM的解码时间对齐（DeAL）。在核心方面，我们将解码视为启发式引导的搜索过程，并促进各种对齐目标的使用。我们的实验显示，通过程序性约束如关键词和长度约束，以及抽象目标如无害和有用，我们可以实现细粒度的权衡并提高对齐目标的遵守程度。最后，我们证明DeAL与现有对齐策略 largely 相辅相成，可以与RLHF和提示技术有效结合，实现更好的对齐。
英文摘要：
Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such as Reinforcement Learning with Human Feedback (RLHF). However, it is unclear if such methods are an effective choice to teach alignment objectives to the model. First, the inability to incorporate multiple, custom rewards and reliance on a model developer's view of universal and static principles are key limitations. Second, the reliability of such approaches is also questionable (e.g. susceptibility to jailbreaking even after safety training). To address these issues, we propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view decoding as a heuristic-guided search process and facilitate the use of a wide variety of alignment objectives. Our experiments with programmatic constraints such as keyword and length constraints, and abstract objectives such as harmlessness and helpfulness, show that we can DeAL with fine-grained trade-offs and improve adherence to alignment objectives. Lastly, we demonstrate that DeAL is largely complementary to existing alignment strategies, and can be effectively paired with RLHF and prompting techniques to achieve better alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
