NeMo-Aligner：用于高效模型对齐的可扩展工具包
NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
AI总结：
本文提出 NeMo-Aligner 工具包，可扩展至千卡 GPU 训练 Nemotron 4 340B 等超大开源模型，高效实现 RLHF、DPO、SteerLM、SPIN 等对齐方法并支持 PEFT，以 Apache 2.0 开源。
摘要：
使大语言模型（LLMs）与人类价值观和偏好对齐，对于让它们变得有用且安全至关重要。然而，构建高效的对齐工具可能具有挑战性，尤其是对于那些通常包含数百亿或数千亿参数的最大、最强的大语言模型。我们创建了 NeMo-Aligner，这是一个模型对齐工具包，能够高效扩展到一千个 GPU，用于训练最大的开源大语言模型，如 Nemotron 4 340B 和 Llama 3.1 405B。NeMo-Aligner 为模型对齐的主要范式提供了高度优化且可扩展的实现，例如：基于人类反馈的强化学习（RLHF）、直接偏好优化（DPO）、SteerLM 和自博弈微调（SPIN）。此外，我们的工具包支持在参数高效微调（PEFT）设置下运行大多数对齐技术。NeMo-Aligner 的设计注重可扩展性，只需极少的工作量即可支持其他对齐技术。它以 Apache 2.0 许可证开源，我们诚邀社区在 https://github.com/NVIDIA/NeMo-Aligner 上贡献代码。
英文摘要：
Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to a thousand GPUs for training the largest open-source LLMs such as Nemotron 4 340B and Llama 3.1 405B. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
