意图因子化生成：释放语言模型的多样性
Intent Factored Generation: Unleashing the Diversity in Your Language Model
AI总结：
本文提出意图因子化生成（IFG），通过先生成语义意图再生成回复并分阶段调节温度，在数学、代码、对话和语言建模任务中提升多样性且保持质量。
摘要：
对于固定提示，从大型语言模型中获得多个具有有意义多样性的高质量样本，仍然是一个开放挑战。当前用于提升多样性的方法通常只在 token 层面操作，即对同一回复进行释义。这会带来问题，因为它导致在推理问题上探索不足，并使对话智能体缺乏吸引力且重复。为解决该问题，我们提出意图因子化生成（Intent Factored Generation，IFG），将采样过程分解为两个阶段。首先，我们采样一个语义密集的意图，例如摘要或关键词。其次，我们以原始提示和第一阶段得到的意图为条件，采样最终回复。这使我们能够在意图步骤中使用较高温度以促进概念多样性，并在最终生成过程中使用较低温度以确保输出连贯且自洽。此外，我们发现，在生成思维链的每一步之前，提示模型明确陈述其对该步骤的意图，对推理任务有益。我们在一组多样化任务上证明了该方法的有效性。结果表明，该方法在数学和代码任务上同时提升了 pass@k 和来自验证器反馈的强化学习（Reinforcement Learning from Verifier Feedback）。在指令微调方面，我们将 IFG 与直接偏好优化（Direct Preference Optimisation）相结合，在不牺牲奖励的情况下增加对话多样性。最后，在通用语言建模任务上，我们使用一个新收集并开源的读者评论与新闻文章数据集，在保持生成质量的同时实现了更高多样性。总之，我们提出了一种在保持性能的同时提升 LLM 样本多样性的简单方法。该方法只需通过更改提示并在生成期间调整温度即可实现，因此易于集成到许多算法中，从而在各类应用中获得收益。
英文摘要：
Obtaining multiple meaningfully diverse, high quality samples from Large Language Models for a fixed prompt remains an open challenge. Current methods for increasing diversity often only operate at the token-level, paraphrasing the same response. This is problematic because it leads to poor exploration on reasoning problems and to unengaging, repetitive conversational agents. To address this we propose Intent Factored Generation (IFG), factorising the sampling process into two stages. First, we sample a semantically dense intent, e.g., a summary or keywords. Second, we sample the final response conditioning on both the original prompt and the intent from the first stage. This allows us to use a higher temperature during the intent step to promote conceptual diversity, and a lower temperature during the final generation to ensure the outputs are coherent and self-consistent. Additionally, we find that prompting the model to explicitly state its intent for each step of the chain-of-thought before generating the step is beneficial for reasoning tasks. We demonstrate our method's effectiveness across a diverse set of tasks. We show this method improves both pass@k and Reinforcement Learning from Verifier Feedback on maths and code tasks. For instruction-tuning, we combine IFG with Direct Preference Optimisation to increase conversational diversity without sacrificing reward. Finally, we achieve higher diversity while maintaining the quality of generations on a general language modelling task, using a new dataset of reader comments and news articles that we collect and open-source. In summary, we present a simple method of increasing the sample diversity of LLMs while maintaining performance. This method can be implemented by changing the prompt and varying the temperature during generation, making it easy to integrate into many algorithms for gains across various applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
