ROSE：一种面向奖励的 LLM 特定任务指令微调数据选择框架
ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning
AI总结：
针对指令微调损失与任务性能不对齐的问题，本文提出 ROSE 框架，利用成对偏好损失和影响函数选择任务相关训练数据，仅用 5% 数据即可媲美全量微调并超越现有方法。
摘要：
指令微调凸显了大型语言模型（LLM）在不同领域生成更易受人类控制且更有效输出方面的巨大潜力。在本研究中，我们关注 LLM 特定任务指令微调的数据选择问题。现有方法主要依赖精心设计的相似性指标，以选择与测试数据分布一致的训练数据。其目标是最小化测试数据上的指令微调损失，最终提升目标任务性能。然而，人们普遍观察到，LLM 中的指令微调损失（即下一 token 预测的交叉熵损失）往往无法与实际任务性能呈现单调关系。这种错位削弱了当前特定任务指令微调数据选择方法的有效性。为解决该问题，我们提出 ROSE，这是一种新颖的面向奖励的指令数据选择方法，利用成对偏好损失作为奖励信号，以优化特定任务指令微调的数据选择。具体而言，ROSE 采用一种影响函数形式，来近似训练数据点相对于少样本偏好验证集的影响，从而选择与任务最相关的训练数据点。实验结果表明，仅使用 ROSE 选择 5% 的训练数据，我们的方法即可取得与使用完整训练数据集微调相比具有竞争力的结果，并且优于其他用于特定任务指令微调的最先进数据选择方法。我们的定性分析进一步证实，该方法在多个基准数据集和多样化模型架构上具有稳健的泛化能力。
英文摘要：
Instruction tuning has underscored the significant potential of large language models (LLMs) in producing more human controllable and effective outputs in various domains. In this work, we focus on the data selection problem for task-specific instruction tuning of LLMs. Prevailing methods primarily rely on the crafted similarity metrics to select training data that aligns with the test data distribution. The goal is to minimize instruction tuning loss on the test data, ultimately improving performance on the target task. However, it has been widely observed that instruction tuning loss (i.e., cross-entropy loss for next token prediction) in LLMs often fails to exhibit a monotonic relationship with actual task performance. This misalignment undermines the effectiveness of current data selection methods for task-specific instruction tuning. To address this issue, we introduce ROSE, a novel Reward-Oriented inStruction data sElection method which leverages pairwise preference loss as a reward signal to optimize data selection for task-specific instruction tuning. Specifically, ROSE adapts an influence formulation to approximate the influence of training data points relative to a few-shot preference validation set to select the most task-related training data points. Experimental results show that by selecting just 5\% of the training data using ROSE, our approach can achieve competitive results compared to fine-tuning with the full training dataset, and it surpasses other state-of-the-art data selection methods for task-specific instruction tuning. Our qualitative analysis further confirms the robust generalizability of our method across multiple benchmark datasets and diverse model architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
