多样性作为奖励：在领域未确定的混合数据上微调大语言模型
Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data
AI总结：
针对领域标签缺失等数据微调LLM的难题，提出赋予LLM双重身份、以多样性为奖励选数据的微调方法，显著提升多任务性能，发布代码并推动数据-模型协同设计。
摘要：
使用多样化数据集微调大语言模型（LLM）对于提升其在各领域的整体表现至关重要。在实际场景中，现有基于数据组成混合比例建模的方法往往难以应对领域标签缺失、不精确或不规范的数据，而基于数据选择的方法通常难以平衡多领域性能。为应对这些挑战，本研究通过实证构建对比数据池并理论推导解释，探究了数据多样性在提升LLM整体能力中的作用。基于所得见解，我们提出一种新方法，赋予LLM双重身份：作为输出模型，基于多样性奖励认知探测并选择数据；同时作为输入模型，用所选数据进行微调。大量实验表明，当应用于多种先进LLM时，该方法在领域未确定数据及一系列基础下游任务上显著提升了性能。我们发布了代码，希望本研究能深化对数据多样性的理解，并推动LLM的反馈驱动数据-模型协同设计。
英文摘要：
Fine-tuning large language models (LLMs) using diverse datasets is crucial for enhancing their overall performance across various domains. In practical scenarios, existing methods based on modeling the mixture proportions of data composition often struggle with data whose domain labels are missing, imprecise or non-normalized, while methods based on data selection usually encounter difficulties in balancing multi-domain performance. To address these challenges, in this work, we investigate the role of data diversity in enhancing the overall abilities of LLMs by empirically constructing contrastive data pools and theoretically deriving explanations. Building upon the insights gained, we propose a new method that gives the LLM a dual identity: an output model to cognitively probe and select data based on diversity reward, as well as an input model to be tuned with the selected data. Extensive experiments show that the proposed method notably boosts performance across domain-undetermined data and a series of foundational downstream tasks when applied to various advanced LLMs. We release our code and hope this study can shed light on the understanding of data diversity and advance feedback-driven data-model co-design for LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
