面向多样化创意写作的大语言模型后训练改进方法
Modifying Large Language Model Post-Training for Diverse Creative Writing
AI总结：
针对大语言模型后训练重质量轻多样性的问题，提出将样本偏差度纳入训练目标的方法，应用于DPO和ORPO可在质量损失极小的前提下提升创意写作输出多样性，8B模型效果经多类验证表现优异。
摘要：
由于创意写作任务不存在唯一正确答案，经训练执行此类任务的大语言模型（LLM）应当能够生成多样的有效输出。然而，LLM后训练通常聚焦于提升生成质量，却忽略了对输出多样性的促进。因此，我们针对创意写作生成场景，研究可同时提升输出多样性与质量的后训练方法。我们的核心思路是将偏差度——即某训练样本与同一提示下所有其他样本的差异程度——纳入训练目标，以促进模型从稀有的高质量样本中学习。我们将该方法应用于直接偏好优化（DPO）和优势比偏好优化（ORPO），实验证明该方法可提升训练后模型的输出多样性，同时仅造成极小的质量下降。我们最优的8B参数模型可达到与人工创建数据集相当的多样性水平，且输出质量与我们测试的最优指令微调模型GPT-4o和DeepSeek-R1相近。我们还通过人工评估、消融实验以及与现有多样化方法DivPO的对比，进一步验证了所提方法的有效性。
英文摘要：
As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diversity. Hence, in creative writing generation, we investigate post-training approaches to promote both output diversity and quality. Our core idea is to include deviation -- the degree of difference between a training sample and all other samples with the same prompt -- in the training objective to facilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimization (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and a comparison to an existing diversification approach, DivPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
