面向自然语言生成的离线强化学习：基于隐式语言Q学习
Offline RL for Natural Language Generation with Implicit Language Q Learning
AI总结：
本研究提出面向自然语言生成的离线强化学习方法ILQL，结合RL效用最大化框架与监督学习优势，通过价值保守性和隐式数据集约束学习价值函数，在对话优化、主观奖励任务上表现更优。
摘要：
大语言模型从文本语料库中提炼出广泛的知识，但在完成用户指定任务时可能表现不一致。这一问题可通过在精选数据集上进行监督学习微调，或通过强化学习微调来解决。在本研究中，我们提出了一种专为语言模型设计的新型离线强化学习方法——隐式语言Q学习（ILQL），该方法结合了强化学习算法灵活的效用最大化框架，以及监督学习利用已收集数据的能力，同时兼具监督学习的简洁性与稳定性。我们的方法在学习价值函数时结合了价值保守性与隐式数据集支持约束，随后利用这些价值函数引导语言模型生成结果，使其最大化用户指定的效用函数。除了对ILQL进行实验验证外，我们还详细实证分析了离线强化学习在自然语言生成场景中的适用情况，证明了在端到端对话任务中，它比现有方法能更有效地优化效用，且能有效优化基于主观判断的高方差奖励函数，例如判断一条评论是否应被标记为有毒评论。
英文摘要：
Large language models distill broad knowledge from text corpora. However, they can be inconsistent when it comes to completing user specified tasks. This issue can be addressed by finetuning such models via supervised learning on curated datasets, or via reinforcement learning. In this work, we propose a novel offline RL method, implicit language Q-learning (ILQL), designed for use on language models, that combines both the flexible utility maximization framework of RL algorithms with the ability of supervised learning to leverage previously collected data, as well as its simplicity and stability. Our method employs a combination of value conservatism alongside an implicit dataset support constraint in learning value functions, which are then used to guide language model generations towards maximizing user-specified utility functions. In addition to empirically validating ILQL, we present a detailed empirical analysis of situations where offline RL can be useful in natural language generation settings, demonstrating how it can be a more effective utility optimizer than prior approaches for end-to-end dialogue, and how it can effectively optimize high variance reward functions based on subjective judgement, such as whether to label a comment as toxic or not.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
