改进基于影响力的指令调优数据选择以实现多样化能力的均衡学习
Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities
AI总结：
该研究针对基于影响力的指令调优数据选择的任务偏向问题，提出BIDS算法，经实验验证其在小比例数据下可实现更均衡性能且优于现有方法。
摘要：
选择合适的训练数据对于大语言模型（LLM）的指令微调至关重要，其目标为：（1）激发强大能力；（2）在不同任务间实现均衡性能。基于影响力的方法有望实现目标（1），通过估算每个训练样本对模型预测的贡献，但往往在实现目标（2）上表现不佳。我们的系统研究表明，这种性能欠佳可归因于固有偏差：部分任务本质上具有更高的影响力。因此，数据选择往往偏向这些任务，不仅损害其他任务的模型性能，且反直觉地，还会损害这些高影响力任务自身的性能。为解决该问题，我们提出BIDS算法，即Balanced and Influential Data Selection（均衡且具影响力的数据选择算法）。BIDS首先对训练数据的影响力得分进行归一化，随后迭代选择对最具代表性不足任务影响力最高的训练样本。在覆盖5种多样化能力的7个基准上，使用Llama-3和Mistral-v0.3开展的实验表明，BIDS始终优于最先进的基于影响力的算法及其他非基于影响力的框架。令人惊讶的是，使用BIDS选择的15%子集进行训练，甚至能以更均衡的性能超越全数据集训练。我们的分析凸显了实例级归一化与所选数据的迭代优化对多样化能力均衡学习的重要性。
英文摘要：
Selecting appropriate training data is crucial for instruction fine-tuning of large language models (LLMs), which aims to (1) elicit strong capabilities, and (2) achieve balanced performance across different tasks. Influence-based methods show promise in achieving (1), by estimating the contribution of each training example to the model's predictions, but often struggle with (2). Our systematic investigation reveals that this underperformance can be attributed to an inherent bias, where some tasks intrinsically have greater influence than others. As a result, data selection is often biased towards these tasks, not only hurting the model's performance on others but also, counterintuitively, harming performance on these high-influence tasks themselves. To address this, we propose BIDS, a Balanced and Influential Data Selection algorithm. BIDS first normalizes influence scores of the training data, and then iteratively chooses the training example with the highest influence on the most underrepresented task. Experiments with both Llama-3 and Mistral-v0.3 on seven benchmarks spanning five diverse capabilities show that BIDS consistently outperforms both state-of-the-art influence-based algorithms and other non-influence-based frameworks. Surprisingly, training on a 15% subset selected by BIDS can even outperform full-dataset training with a much more balanced performance. Our analysis highlights the importance of both instance-level normalization and iterative optimization of selected data for balanced learning of diverse capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
