CodecLM：利用定制合成数据对齐语言模型
CodecLM: Aligning Language Models with Tailored Synthetic Data
AI总结：
CodecLM框架通过编码-解码原理，利用LLM将种子指令编码为元数据并解码生成定制合成数据，结合Self-Rubrics和对比过滤，在多个基准上超越现有方法，提升指令遵循能力。
摘要：
指令微调已成为将大语言模型（LLMs）与特定任务指令对齐的关键手段，从而缓解下一词元预测目标与用户实际目标之间的偏差。为了降低人工收集或标注数据的劳动和时间成本，研究者开始探索利用LLMs生成指令对齐的合成数据。近期工作侧重于生成多样化指令并利用LLM提高指令复杂度，但往往忽视下游使用场景。目前仍不清楚如何定制高质量数据，以在不同的目标指令分布和LLMs中激发更好的指令遵循能力。为此，我们提出了CodecLM，这是一个通用框架，用于自适应地生成高质量合成数据，以实现LLM与不同下游指令分布和LLMs的对齐。借鉴编码-解码原理，我们将LLMs用作编解码器来引导数据生成过程。我们首先将种子指令编码为元数据，这些元数据是即时生成的简洁关键词，用于捕捉目标指令分布，然后解码元数据以创建定制指令。我们还在解码过程中引入了Self-Rubrics和Contrastive Filtering，以定制数据高效样本。在四个开放域指令遵循基准上的大量实验验证了CodecLM相对于当前最先进方法的有效性。
英文摘要：
Instruction tuning has emerged as the key in aligning large language models (LLMs) with specific task instructions, thereby mitigating the discrepancy between the next-token prediction objective and users' actual goals. To reduce the labor and time cost to collect or annotate data by humans, researchers start to explore the use of LLMs to generate instruction-aligned synthetic data. Recent works focus on generating diverse instructions and applying LLM to increase instruction complexity, often neglecting downstream use cases. It remains unclear how to tailor high-quality data to elicit better instruction-following abilities in different target instruction distributions and LLMs. To this end, we introduce CodecLM, a general framework for adaptively generating high-quality synthetic data for LLM alignment with different downstream instruction distributions and LLMs. Drawing on the Encode-Decode principles, we use LLMs as codecs to guide the data generation process. We first encode seed instructions into metadata, which are concise keywords generated on-the-fly to capture the target instruction distribution, and then decode metadata to create tailored instructions. We also introduce Self-Rubrics and Contrastive Filtering during decoding to tailor data-efficient samples. Extensive experiments on four open-domain instruction following benchmarks validate the effectiveness of CodecLM over the current state-of-the-arts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
