Mind the Gap：通过 Conformative Decoding 提升指令微调大型语言模型的输出多样性
Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models
AI总结：
本文针对指令微调导致 LLM 输出多样性下降的问题，分析 OLMo 系列各微调阶段并发现 DPO 影响最大，进而提出 conformative decoding，用基座模型引导指令模型以恢复多样性并保持质量。
摘要：
对大型语言模型（LLM）进行指令微调会降低其输出多样性，这会影响许多任务，尤其是创意任务。本文研究写作提示叙事生成任务中的“多样性差距”。对于多种开放权重和开源 LLM，当前多样性指标都测得这一差距。结果表明，指令微调导致多样性显著下降。为进一步理解输出多样性如何受到影响，我们考察了 OLMo 和 OLMo 2 模型在每个微调阶段的多样性损失。结果显示，DPO 对多样性的影响最大。受这些发现启发，我们提出一种新的解码策略 conformative decoding，它利用多样性更高的基座模型来引导指令模型，从而重新引入输出多样性。我们表明，conformative decoding 通常能够提升多样性，甚至保持或改善质量。
英文摘要：
Instruction-tuning large language models (LLMs) reduces the diversity of their outputs, which has implications for many tasks, particularly for creative tasks. This paper investigates the ``diversity gap'' for a writing prompt narrative generation task. This gap emerges as measured by current diversity metrics for various open-weight and open-source LLMs. The results show significant decreases in diversity due to instruction-tuning. We explore the diversity loss at each fine-tuning stage for the OLMo and OLMo 2 models to further understand how output diversity is affected. The results indicate that DPO has the most substantial impact on diversity. Motivated by these findings, we present a new decoding strategy, conformative decoding, which guides an instruct model using its more diverse base model to reintroduce output diversity. We show that conformative decoding typically increases diversity and even maintains or improves quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
