RLHFSpec: 通过自适应草稿打破RLHF训练的效率瓶颈
RLHFSpec: Breaking the Efficiency Bottleneck in RLHF Training via Adaptive Drafting
AI总结：
RLHFSpec通过自适应草稿策略和高效样本再分配，提升RLHF训练效率，缓解生成瓶颈，提高整体性能。
摘要：
强化学习从人类反馈（RLHF）是大型语言模型（LLMs）的重要微调技术，包含三个阶段：生成、推理和训练。生成阶段生成样本，然后用于推断可学习的经验以进行训练。我们观察到生成阶段是整个执行过程的瓶颈，并将其视为优化的关键点。具体来说，我们首次尝试将推测解码整合到RLHF生成阶段，并提出RLHFSpec，一种通过高效推测解码和样本再分配加速生成执行的RLHF系统。为了充分利用推测解码提供的性能潜力，尤其是处理生成阶段的动态工作负载，RLHFSpec提出了一种工作负载感知的草稿策略选择机制，该机制通过联合考虑验证成本和接受的token数量，选择近最优的策略。此外，RLHFSpec还提出样本再分配，以充分利用GPU资源，并通过高效的样本迁移机制进行优化。实验结果表明，RLHFSpec在生成阶段相比最先进的方法能够实现更高的吞吐量。此外，由于有效缓解了生成瓶颈，RLHFSpec在整个RLHF执行过程中也显示出显著的性能加速。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) is an important fine-tuning technique for large language models (LLMs) and comprises three stages: generation, inference, and training. The generation stage generates samples that are then used to infer learnable experiences for training. We observe that the generation stage is the bottleneck of the entire execution process and consider it a key point for optimization. Specifically, we realize the first attempt to integrate speculative decoding into the RLHF generation stage and propose RLHFSpec, an RLHF system that accelerates generation execution with efficient speculative decoding and sample reallocation. To fully exploit the performance potential provided by speculative decoding, especially dealing with the dynamic workload of the generation stage, RLHFSpec proposes a workload-aware drafting strategy selection mechanism, which selects the near-optimal strategy by jointly considering the verification cost and the number of accepted tokens. Moreover, RLHFSpec also proposes sample reallocation to fully utilize the GPU resources, and optimizes it with an efficient sample migration mechanism. The experimental results show that the RLHFSpec can achieve higher throughput in the generation stage compared to state-of-the-art works. Moreover, due to the effective alleviation of the generation bottleneck, RLHFSpec also shows significant performance speedup in the entire RLHF execution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
