Group Preference Optimization：大型语言模型的少样本对齐
Group Preference Optimization: Few-Shot Alignment of Large Language Models
AI总结：
本文提出 Group Preference Optimization（GPO）框架，通过元学习训练独立的上下文自回归 transformer 模块，以少样本方式实现大型语言模型对不同群体偏好的高效精准对齐，显著降低数据和计算资源消耗。
摘要：
大型语言模型（LLMs）的许多应用（从聊天机器人到创意写作）都需要细致的主观判断，而这些判断在不同群体之间可能存在显著差异。现有的对齐算法为每个群体进行对齐的成本可能很高，需要大量特定群体的偏好数据和计算，这在实际应用场景中是难以承受的。我们引入了 Group Preference Optimization（GPO），这是一个以少样本方式引导语言模型适应各个群体偏好的对齐框架。在 GPO 中，我们用一个独立的 transformer 模块来增强基础 LLM，该模块经过训练，用于预测群体对 LLM 生成内容的偏好。为了实现少样本学习，我们将该模块参数化为一个上下文自回归 transformer，并通过在多个群体上进行元学习来训练它。我们通过在不同规模的 LLM 上进行严格评估，在三项人类意见适应任务中实证验证了 GPO 的有效性。这些任务涉及适应美国人口群体、全球国家以及个人用户的偏好。我们的结果表明，GPO 不仅能更准确地对齐模型，而且需要更少的特定群体偏好数据，以及更少的训练和推理计算资源，其表现优于上下文引导和微调方法等现有策略。
英文摘要：
Many applications of large language models (LLMs), ranging from chatbots to creative writing, require nuanced subjective judgments that can differ significantly across different groups. Existing alignment algorithms can be expensive to align for each group, requiring prohibitive amounts of group-specific preference data and computation for real-world use cases. We introduce Group Preference Optimization (GPO), an alignment framework that steers language models to preferences of individual groups in a few-shot manner. In GPO, we augment the base LLM with an independent transformer module trained to predict the preferences of a group for the LLM generations. For few-shot learning, we parameterize this module as an in-context autoregressive transformer and train it via meta-learning on several groups. We empirically validate the efficacy of GPO through rigorous evaluations using LLMs with varied sizes on three human opinion adaptation tasks. These tasks involve adapting to the preferences of US demographic groups, global countries, and individual users. Our results demonstrate that GPO not only aligns models more accurately but also requires fewer group-specific preferences, and less training and inference computing resources, outperforming existing strategies such as in-context steering and fine-tuning methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
