解锁解码时可控性：基于对比提示的无梯度多目标对齐
Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Prompts
AI总结：
针对多目标对齐需训练多模型且扩展性差的问题，提出无梯度 MCA，在解码时用专家与对抗提示的对比组合平衡目标，并获得更优 Pareto 前沿。
摘要：
多目标对齐任务旨在平衡和控制大型语言模型的不同对齐目标（例如有用性、无害性和诚实性），以满足不同用户的个性化需求。然而，以往方法往往需要训练多个模型来处理各种用户偏好，训练模型的数量会随着对齐目标数量和不同偏好数量呈线性增长。同时，现有方法通常可扩展性较差，每考虑一个新的对齐目标都需要进行大量重新训练。鉴于以往方法的局限，我们提出 MCA（Multi-objective Contrastive Alignemnt，多目标对比对齐），该方法为每个目标构建专家提示和对抗提示，在解码时进行对比，并通过组合这些对比来平衡各目标。实验验证表明，我们的方法在不同对齐目标之间获得分布良好的 Pareto 前沿方面优于以往方法。
英文摘要：
The task of multi-objective alignment aims at balancing and controlling the different alignment objectives (e.g., helpfulness, harmlessness and honesty) of large language models to meet the personalized requirements of different users. However, previous methods tend to train multiple models to deal with various user preferences, with the number of trained models growing linearly with the number of alignment objectives and the number of different preferences. Meanwhile, existing methods are generally poor in extensibility and require significant re-training for each new alignment objective considered. Considering the limitation of previous approaches, we propose MCA (Multi-objective Contrastive Alignemnt), which constructs an expert prompt and an adversarial prompt for each objective to contrast at the decoding time and balances the objectives through combining the contrast. Our approach is verified to be superior to previous methods in obtaining a well-distributed Pareto front among different alignment objectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
