Stream Aligner：通过分布归纳实现高效的句子级对齐
Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
AI总结：
本文提出 Stream Aligner，利用小模型归纳后续句子偏好分布并进行迭代句子级替换，以更低延迟和额外模型依赖提升 LLM 对齐、推理及数学能力。
摘要：
大型语言模型（LLM）的快速发展使其能力取得了显著提升，但也加剧了人们对其与人类价值观和意图对齐的担忧。当前的对齐策略（包括自适应训练和推理时方法）已在该领域展现出潜力。然而，这些方法仍难以在不同任务和难度之间平衡部署复杂性与能力。在本工作中，我们提出了 Streaming Distribution Induce Aligner（Stream Aligner），这是一种新颖的对齐范式，在整个生成过程中的各类任务上兼具效率与增强的性能。Stream Aligner 通过使用一个小模型学习后续句子的偏好，迭代纠正上游模型输出的后续句子，然后使用纠正后的句子替换后续生成中的该句子，从而实现动态的句子级纠正。实验表明，与 Aligner 相比，Stream Aligner 减少了对额外模型能力的依赖，增强了 LLM 的推理能力，并降低了用户交互期间的延迟。具体而言，Stream Aligner-2B 模型在受测 Llama2-70B-chat 模型上实现了有用性 76.1%、无害性 36.0% 的提升；Stream Aligner-8B 在受测 Llama3-70B-Instruct 模型的数学能力上实现了 3.5% 的提升。
英文摘要：
The rapid advancement of large language models (LLMs) has led to significant improvements in their capabilities, but also to increased concerns about their alignment with human values and intentions. Current alignment strategies, including adaptive training and inference-time methods, have demonstrated potential in this area. However, these approaches still struggle to balance deployment complexity and capability across various tasks and difficulties. In this work, we introduce the Streaming Distribution Induce Aligner (Stream Aligner), a novel alignment paradigm that combines efficiency with enhanced performance in various tasks throughout the generation process. Stream Aligner achieves dynamic sentence-level correction by using a small model to learn the preferences of the suffix sentence, iteratively correcting the suffix sentence output by the upstream model, and then using the corrected sentence to replace the suffix sentence in subsequent generations. Compared to Aligner, our experiments demonstrate that Stream Aligner reduces reliance on the capabilities of additional models, enhances the reasoning abilities of LLMs, and decreases latency during user interaction. Specifically, Stream Aligner-2B model has achieved an improvement of 76.1% in helpfulness, 36.0% in harmlessness on the tested Llama2-70B-chat model, and Stream Aligner-8B has achieved an improvement of 3.5% on the math ability of the tested Llama3-70B-Instruct model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
