Mixture-of-Skills：学习优化大语言模型微调的数据使用
Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models
AI总结：
针对大语言模型微调中数据集异构不平衡导致的技能发展难平衡的问题，提出通用强化学习框架MoS动态优化数据使用，还推出适配特定任务的MoSpec，经多骨干多基准实验验证可显著提升模型性能。
摘要：
大语言模型（LLM）通常在来源各异的多样化大规模数据集上进行微调，以培养写作、推理、聊天、编码等全面的技能。每种技能都有其独特特征，且这些数据集往往具有异构性和不平衡性，这使得微调过程极具挑战性。要平衡各项技能的发展同时确保模型维持整体性能，需要复杂的技术和精心的数据集整理。在本研究中，我们提出了一种通用的、与模型无关的强化学习框架Mixture-of-Skills（MoS，技能混合），该框架可在微调过程中自动学习优化数据使用。该框架通过根据不同数据集当前的学习状态动态调整对其的关注程度，确保大语言模型实现最优的综合技能发展。为验证MoS的有效性，我们使用三种不同的大语言模型骨干网络在两个广泛使用的基准上开展了大量实验，结果表明MoS显著提升了模型性能。在MoS取得成功的基础上，我们提出了MoSpec，这是一种针对特定任务微调的适配方案，可利用各类数据集的效用以实现特定目标。我们的研究凸显了数据集再平衡的重要意义，并表明MoS是一种强大的通用解决方案，可用于优化各类用途下大语言模型微调的数据使用。
英文摘要：
Large language models (LLMs) are typically fine-tuned on diverse and extensive datasets sourced from various origins to develop a comprehensive range of skills, such as writing, reasoning, chatting, coding, and more. Each skill has unique characteristics, and these datasets are often heterogeneous and imbalanced, making the fine-tuning process highly challenging. Balancing the development of each skill while ensuring the model maintains its overall performance requires sophisticated techniques and careful dataset curation. In this work, we propose a general, model-agnostic, reinforcement learning framework, Mixture-of-Skills (MoS), that learns to optimize data usage automatically during the fine-tuning process. This framework ensures the optimal comprehensive skill development of LLMs by dynamically adjusting the focus on different datasets based on their current learning state. To validate the effectiveness of MoS, we conduct extensive experiments using three diverse LLM backbones on two widely used benchmarks and demonstrate that MoS substantially enhances model performance. Building on the success of MoS, we propose MoSpec, an adaptation for task-specific fine-tuning, which harnesses the utilities of various datasets for a specific purpose. Our work underlines the significance of dataset rebalancing and present MoS as a powerful, general solution for optimizing data usage in the fine-tuning of LLMs for various purposes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
