自动化数据整理用于稳健的大语言模型微调
Automated Data Curation for Robust Language Model Fine-Tuning
AI总结：
提出CLEAR自动化数据整理流程，基于置信度过滤或修正低质量训练数据，无需额外微调计算，即可提升任何大语言模型微调后的性能。
摘要：
大语言模型已成为序列到序列文本生成任务的事实标准方法，但对于专门的任务/领域，预训练的大语言模型缺乏生成准确或格式良好响应的特定能力。监督微调通过在包含示例提示和目标响应的数据集上训练来使大语言模型专业化，但现实世界的数据往往含有噪声。虽然存在许多微调算法，这里我们从数据中心的AI视角考虑大语言模型微调，研究如何系统地整理训练数据集，以改进通过任何微调算法产生的大语言模型。我们引入了一个自动化的数据整理流程CLEAR（基于置信度的大语言模型评估与修正），用于指令调优数据集，该流程可与任何大语言模型和微调程序一起使用。CLEAR估计哪些训练数据质量低，并对其进行过滤或修正。通过大语言模型派生的置信度估计来自动识别要过滤或修正的数据，以确保仅对数据集进行有信心的修改。与现有的数据整理技术不同，CLEAR是一个全面的框架，可以在无需额外微调计算的情况下改进数据集（以及训练模型的输出）。我们不假设能访问比正在微调的模型更强的大语言模型（例如，在微调GPT-3.5时依赖GPT-4），以观察CLEAR是否能有意义地提升任何大语言模型的能力。实验表明，CLEAR在多个数据集和模型（如GPT-3.5和Llama2）上持续提高了微调模型的性能。
英文摘要：
Large Language Models have become the de facto approach to sequence-to-sequence text generation tasks, but for specialized tasks/domains, a pretrained LLM lacks specific capabilities to produce accurate or well-formatted responses. Supervised fine-tuning specializes a LLM by training it on dataset of example prompts with target responses, but real-world data tends to be noisy. While many fine-tuning algorithms exist, here we consider a \emph{data-centric AI} perspective on LLM fine-tuning, studying how to \emph{systematically} curate the training dataset to improve the LLM produced via \emph{any} fine-tuning algorithm. We introduce an automated data curation pipeline CLEAR (Confidence-based LLM Evaluation And Rectification) for instruction tuning datasets, that can be used with any LLM and fine-tuning procedure. CLEAR estimates which training data is low-quality and either filters or corrects it. Automatically identifying which data to filter or correct is done via LLM-derived confidence estimates, to ensure only confident modifications to the dataset. Unlike existing data curation techniques, CLEAR is a comprehensive framework that can improve a dataset (and trained model outputs) without additional fine-tuning computations. We don't assume access to a stronger LLM than the model being fine-tuned (e.g.\ relying on GPT-4 when fine-tuning GPT-3.5), to see whether CLEAR can meaningfully improve the capabilities of any LLM. Experiments reveal that CLEAR consistently improves the performance of fine-tuned models across many datasets and models (like GPT-3.5 and Llama2).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
