Collab：使用多智能体混合的受控解码用于大语言模型对齐
Collab: Controlled Decoding using Mixture of Agents for LLM Alignment
AI总结：
提出Collab，一种基于多智能体混合的受控解码方法，在推理时通过令牌级动态选择最优LLM实现对齐，理论保证最优性能，实验显示平均奖励提升1.56倍，胜平率71.89%。
摘要：
大语言模型（LLMs）的对齐对于在应用中安全可信的部署至关重要。基于人类反馈的强化学习（RLHF）已成为将LLMs与人类偏好和更广泛的效用对齐的有效技术，但它需要更新数十亿模型参数，计算成本高昂。相比之下，受控解码提供了一种在推理时对齐模型而无需重新训练的机制。然而，由于任务的复杂性和多变性，单智能体解码方法往往难以适应多样化的任务。为了增强针对目标任务在测试时的性能，我们提出了一种基于多智能体混合的解码策略，利用现有的现成对齐LLM策略。将每个先验策略视为智能体，本着多智能体协作的精神，我们开发了一种解码方法，通过多个智能体之间的令牌级选择策略实现推理时对齐。对于每个令牌，基于长期效用指标从模型池中动态选择最合适的LLM。这种策略切换机制确保了每一步的最优模型选择，使LLMs在解码过程中能够高效协作和对齐。我们提出算法的理论分析确立了对于给定现成模型，以目标奖励表示的目标任务的最优性能。我们使用开源对齐模型在各种任务和偏好上进行了全面的实证评估，证明了该方法相对于单智能体解码基线的优势。值得注意的是，Collab超越了当前最先进的解码策略，平均奖励提升了高达1.56倍，基于GPT-4的胜平率达到71.89%。
英文摘要：
Alignment of Large Language models (LLMs) is crucial for safe and trustworthy deployment in applications. Reinforcement learning from human feedback (RLHF) has emerged as an effective technique to align LLMs to human preferences and broader utilities, but it requires updating billions of model parameters, which is computationally expensive. Controlled Decoding, by contrast, provides a mechanism for aligning a model at inference time without retraining. However, single-agent decoding approaches often struggle to adapt to diverse tasks due to the complexity and variability inherent in these tasks. To strengthen the test-time performance w.r.t the target task, we propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies. Treating each prior policy as an agent in the spirit of mixture of agent collaboration, we develop a decoding method that allows for inference-time alignment through a token-level selection strategy among multiple agents. For each token, the most suitable LLM is dynamically chosen from a pool of models based on a long-term utility metric. This policy-switching mechanism ensures optimal model selection at each step, enabling efficient collaboration and alignment among LLMs during decoding. Theoretical analysis of our proposed algorithm establishes optimal performance with respect to the target task represented via a target reward for the given off-the-shelf models. We conduct comprehensive empirical evaluations with open-source aligned models on diverse tasks and preferences, which demonstrates the merits of this approach over single-agent decoding baselines. Notably, Collab surpasses the current SoTA decoding strategy, achieving an improvement of up to 1.56x in average reward and 71.89% in GPT-4 based win-tie rate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
