通过对比解码合成指令微调数据集
Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
AI总结：
本文提出CoDIT方法，通过对比解码分离预训练知识与指令遵循能力，提升指令微调效果。实验表明基于CoDIT构建的数据集在多个基准上表现更优。
摘要：
使用高性能大语言模型（LLM）生成的响应进行指令微调已成为普遍方法。然而现有文献忽视了LLM响应的一个特性：它们将预训练获得的世界知识与后训练获得的指令遵循能力混淆。我们假设分离指令遵循能力与预训练知识能提升指令微调效果。为此，我们提出CoDIT，一种在响应生成过程中对比解码后训练模型与其预训练 counterpart 的方法。该方法抑制两个模型间共享的预训练知识，同时放大通过后训练获得的指令遵循行为，从而生成更纯粹反映指令遵循能力的响应。实验结果表明，基于CoDIT构建的数据集训练的模型在多个基准上均优于直接生成响应的模型。此外，我们理论和实证表明CoDIT可被视为将参数空间中的聊天向量映射到文本空间，使不同架构模型间的指令微调能力转移成为可能。
英文摘要：
Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
