Meta-Aligner: 多目标大语言模型对齐的双向偏好-策略优化
Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
AI总结：
本文提出Meta-Aligner框架，通过双向优化偏好与策略响应，生成动态偏好以提升训练稳定性，实验证明其在多目标基准上的优越性能。
摘要：
多目标对齐旨在通过同时优化多个目标来对齐大语言模型（LLMs）与多样且常冲突的人类价值观。现有方法主要依赖静态偏好权重构建策略。然而，严格对齐固定目标会丢弃有价值的信息，因为训练响应本质上包含有效的偏好权衡，即使偏离目标。为此，我们提出Meal（MEta ALigner），即双层元学习框架，实现偏好与策略响应之间的双向优化，生成具有指导性的动态偏好以稳定训练。具体而言，我们引入偏好权重网络作为元学习器，根据输入提示生成适应性偏好权重，并将其作为可学习参数更新；同时，LLM策略作为基础学习器，在这些偏好条件下优化响应生成，采用拒绝采样策略。大量实验证明，我们的方法在多个多目标基准上实现了优越性能，验证了动态双向偏好-策略优化框架的有效性。
英文摘要：
Multi-Objective Alignment aims to align Large Language Models (LLMs) with diverse and often conflicting human values by optimizing multiple objectives simultaneously. Existing methods predominantly rely on static preference weight construction strategies. However, rigidly aligning to fixed targets discards valuable intermediate information, as training responses inherently embody valid preference trade-offs even when deviating from the target. To address this limitation, we propose Meal, i.e., MEta ALigner, a bi-level meta-learning framework enabling bidirectional optimization between preferences and policy responses, generating instructive dynamic preferences for steadier training. Specifically, we introduce a preference-weight-net as a meta-learner to generate adaptive preference weights based on input prompts and update the preference weights as learnable parameters, while the LLM policy acts as a base-learner optimizing response generation conditioned on these preferences with rejection sampling strategy. Extensive empirical results demonstrate that our method achieves superior performance on several multi-objective benchmarks, validating the effectiveness of the dynamic bidirectional preference-policy optimization framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
