MetaAligner：迈向语言模型的可泛化多目标对齐
MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models
AI总结：
本文提出策略无关且可泛化的多目标对齐方法MetaAligner，通过动态目标重构、条件弱到强校正和可泛化推理三阶段，在多个策略模型上实现均衡对齐改进并大幅降低训练成本。
摘要：
近年来，大语言模型（LLMs）的进展集中于通过多目标偏好对齐来对齐异构的人类期望与价值观。然而，现有方法依赖于策略模型参数，这要求对每个新策略模型高成本地重复其对齐算法，并且由于其静态对齐目标，它们无法扩展到未见过的目标。在本工作中，我们提出了元目标对齐器（MetaAligner），这是首个策略无关且可泛化的多目标偏好对齐方法。MetaAligner将多目标对齐建模为三个阶段：（1）动态目标重构算法重新组织传统对齐数据集，以监督模型在不同目标上执行灵活对齐；（2）条件弱到强校正范式将固定策略模型的弱输出对齐到在相应对齐目标中具有更高偏好的强输出，从而能够在任何策略模型上实现即插即用的推理，这显著降低了训练成本，并有助于对闭源策略模型进行对齐；（3）可泛化推理方法通过更新提示中的目标文本描述来灵活调整目标，从而促进对未见目标的可泛化对齐。实验结果表明，MetaAligner在10个最先进的策略模型上实现了显著且均衡的多目标对齐改进，并且与之前的对齐方法相比，节省了高达93.63%的GPU训练小时数。该模型还能有效对齐未见过的目标，标志着迈向可泛化多目标偏好对齐的第一步。
英文摘要：
Recent advancements in large language models (LLMs) focus on aligning to heterogeneous human expectations and values via multi-objective preference alignment. However, existing methods are dependent on the policy model parameters, which require high-cost repetition of their alignment algorithms for each new policy model, and they cannot expand to unseen objectives due to their static alignment objectives. In this work, we propose Meta-Objective Aligner (MetaAligner), the first policy-agnostic and generalizable method for multi-objective preference alignment. MetaAligner models multi-objective alignment into three stages: (1) dynamic objectives reformulation algorithm reorganizes traditional alignment datasets to supervise the model on performing flexible alignment across different objectives; (2) conditional weak-to-strong correction paradigm aligns the weak outputs of fixed policy models to approach strong outputs with higher preferences in the corresponding alignment objectives, enabling plug-and-play inferences on any policy models, which significantly reduces training costs and facilitates alignment on close-source policy models; (3) generalizable inference method flexibly adjusts target objectives by updating their text descriptions in the prompts, facilitating generalizable alignment to unseen objectives. Experimental results show that MetaAligner achieves significant and balanced improvements in multi-objective alignments on 10 state-of-the-art policy models, and saves up to 93.63% of GPU training hours compared to previous alignment methods. The model also effectively aligns unseen objectives, marking the first step towards generalizable multi-objective preference alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
