基于Holdout损失的数据选择用于LLM微调的上下文学习
Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
AI总结：
本文提出基于上下文近似（ICA）的方法，通过估计holdout损失来高效选择高价值训练数据，提升LLM微调的对齐性能。
摘要：
对大型预训练语言模型进行微调是使它们与人类偏好对齐的常见方法，但噪声或偏离目标的例子会稀释监督。虽然小而精心挑选的数据集往往能与远大于其的数据集匹配性能，但系统且高效地识别高价值训练数据的方法仍处于探索阶段。许多当前方法依赖于启发式或昂贵的重新训练。我们提出了一种原则性、资源高效的框架用于数据选择和重新加权。其核心是一种上下文近似（ICA），它通过在上下文中对小而精心挑选的holdout集进行条件来估计模型在训练候选示例后所承受的holdout损失。ICA不需要参考模型也不需要额外的微调。我们定义由此得到的估计值为ICA分数，并推导出每个示例的权重，这些权重会随着模型参数的变化动态地重新加权梯度更新。在SFT、DPO和SimPO以及多样化的backbone和数据集上，基于ICA的重新加权一致地提高了模型对齐性，且具有最小的开销。我们分析了对分数更新频率和上下文holdout示例数量的敏感性。我们还讨论了在快速漂移的on-policy设置中的局限性，突显了未来工作的方向。代码和提示将被发布。
英文摘要：
Fine-tuning large pretrained language models is a common approach for aligning them with human preferences, but noisy or off-target examples can dilute supervision. While small, well-chosen datasets often match the performance of much larger ones, systematic and efficient ways to identify high-value training data remain underexplored. Many current methods rely on heuristics or expensive retraining. We present a principled, resource-efficient framework for data selection and reweighting. At its core is an In-Context Approximation (ICA) that estimates the holdout loss a model would incur after training on a candidate example by conditioning on a small, curated holdout set in context. ICA requires no reference model and no additional finetuning. We define the resulting estimate as the ICA score, and derive per-example weights that dynamically reweight gradient updates as model parameters evolve. Across SFT, DPO, and SimPO, and over diverse backbones and datasets, ICA-based reweighting consistently improves model alignment with minimal overhead. We analyze sensitivity to score update frequency and the number of in-context holdout examples. We also discuss limitations in rapidly drifting on-policy settings, highlighting directions for future work. Code and prompts will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
