TeaMs-RL：通过强化学习教LLM生成更好的指令数据集
TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
AI总结：
TeaMs-RL通过强化学习直接生成多样化的基础指令数据集用于微调，减少人工参与和外部模型查询，在提升LLM复杂指令处理能力的同时增强隐私保护。
摘要：
大型语言模型（LLM）的发展经常面临诸多挑战，这些挑战源于在人类反馈强化学习（RLHF）框架中严重依赖人类标注员，或是自指令（self-instruct）范式中频繁且昂贵的外部查询。在这项工作中，我们转向强化学习（RL）——但有所不同。与典型的RLHF在指令数据训练后改进LLM不同，我们使用RL直接生成基础指令数据集，该数据集足以用于微调。我们的方法TeaMs-RL使用一套文本操作和规则，优先实现训练数据集的多样化。它促进了高质量数据的生成，而无需过度依赖外部高级模型，为单一微调步骤铺平了道路，并消除了后续RLHF阶段的需要。我们的发现突出了该方法的关键优势：减少了人工参与和模型查询（仅为强基线总量的5.73%），与强基线相比，增强了LLM在制定和理解复杂指令方面的能力，并大幅提升了模型隐私保护。代码可在以下链接获取：https://github.com/SafeRL-Lab/TeaMs-RL
英文摘要：
The development of Large Language Models (LLMs) often confronts challenges stemming from the heavy reliance on human annotators in the reinforcement learning with human feedback (RLHF) framework, or the frequent and costly external queries tied to the self-instruct paradigm. In this work, we pivot to Reinforcement Learning (RL) -- but with a twist. Diverging from the typical RLHF, which refines LLMs following instruction data training, we use RL to directly generate the foundational instruction dataset that alone suffices for fine-tuning. Our method, TeaMs-RL, uses a suite of textual operations and rules, prioritizing the diversification of training datasets. It facilitates the generation of high-quality data without excessive reliance on external advanced models, paving the way for a single fine-tuning step and negating the need for subsequent RLHF stages. Our findings highlight key advantages of our approach: reduced need for human involvement and fewer model queries (only 5.73% of the strong baseline's total), along with enhanced capabilities of LLMs in crafting and comprehending complex instructions compared to strong baselines, and substantially improved model privacy protection. Code is available at the link: https://github.com/SafeRL-Lab/TeaMs-RL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
