随着大语言模型（LLM）的快速发展以及实际需求的不断演进，寻找一种高效且有效的对齐方法变得前所未有的重要。然而，当前对齐方法的复杂性与部署场景中快速迭代的需求之间存在矛盾，这就要求开发一种能够在这些约束条件下运行的模型无关对齐方法。本文提出了Aligner，这是一种新颖且简洁的对齐范式，它使用小型模型学习偏好答案与非偏好答案之间的校正残差。Aligner被设计为一个模型无关的即插即用模块，仅需一次性训练即可直接应用于各种开源模型和基于API的模型，适用于快速迭代场景。值得注意的是，Aligner可应用于任何强大的大规模上游模型。此外，它甚至可以将校正后的响应作为合成人类偏好数据，对上游模型进行迭代式自举提升，突破模型的性能上限。我们的实验表明，在3H维度（有用性、无害性、诚实性）上进行评估，将同一个Aligner模型部署在11种不同的LLM上均能实现性能提升。具体而言，Aligner-7B在测试的所有LLM中实现了有用性平均提升68.9%、无害性平均提升23.8%的效果，同时有效减少了幻觉。在Alpaca-Eval排行榜上，将Aligner-2B堆叠在GPT-4 Turbo上，使其LC胜率从55.0%提升至58.3%，超过了GPT-4 Omni 57.5%的胜率（社区报告数据）。
英文摘要：
With the rapid development of large language models (LLMs) and ever-evolving practical requirements, finding an efficient and effective alignment method has never been more critical. However, the tension between the complexity of current alignment methods and the need for rapid iteration in deployment scenarios necessitates the development of a model-agnostic alignment approach that can operate under these constraints. In this paper, we introduce Aligner, a novel and simple alignment paradigm that learns the correctional residuals between preferred and dispreferred answers using a small model. Designed as a model-agnostic, plug-and-play module, Aligner can be directly applied to various open-source and API-based models with only one-off training, making it suitable for rapid iteration. Notably, Aligner can be applied to any powerful, large-scale upstream models. Moreover, it can even iteratively bootstrap the upstream models using corrected responses as synthetic human preference data, breaking through the model's performance ceiling. Our experiments demonstrate performance improvements by deploying the same Aligner model across 11 different LLMs, evaluated on the 3H dimensions (helpfulness, harmlessness, and honesty). Specifically, Aligner-7B has achieved an average improvement of 68.9% in helpfulness and 23.8% in harmlessness across the tested LLMs while also effectively reducing hallucination. In the Alpaca-Eval leaderboard, stacking Aligner-2B on GPT-4 Turbo improved its LC Win Rate from 55.0% to 58.3%, surpassing GPT-4 Omni's 57.5% Win Rate (community report).
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Research（IBM研究院）；University of Waterloo（滑铁卢大学）；University of Michigan（密歇根大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
Institute for AI, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, Institute for AI, Peking University（北京大学人工智能研究院通用人工智能国家重点实验室）
CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment
School of Computer Science and Technology, Xinjiang University, China（新疆大学计算机科学与技术学院）；Speech Processing and Machine Intelligence (SPMI) Lab, Tsinghua University, China（清华大学语音处理与机器智能实验室）
