arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.04181cs.AIcs.CL
收藏

编排具有不同个性化的大语言模型

Orchestrating LLMs with Different Personalizations

Cornell University(康奈尔大学)

Jin Peng Zhou, Katie Z Luo, Jingwen Gu, Jason Yuan, Kilian Q. Weinberger, Wen Sun

AI总结:

本文提出一种无需重新训练即可按多维度个性化偏好生成文本的方法,通过轻量级偏好控制模型在token级别动态合并多个专家LLM的输出,实验证明其效果匹配或优于现有偏好合并技术。

摘要:

本文提出了一种使大语言模型(LLMs)与个体人类偏好对齐的新方法,该方法有时被称为基于个性化人类反馈的强化学习(RLPHF)。给定在多个维度上声明的偏好,例如有用性、简洁性或幽默感,目标是在不重新训练的情况下创建一个最符合该规格的LLM。从专门的专家LLM出发,每个专家LLM针对一个特定的偏好维度进行训练,我们提出了一种黑盒方法,在逐token级别上合并它们的输出。我们训练了一个轻量级的偏好控制模型(PCM),该模型动态地将偏好描述和当前上下文转化为下一个token的预测权重。通过在token级别组合专家模型的输出,我们的方法能够动态生成优化给定偏好的文本。实证测试表明,我们的方法匹配或超越了现有的偏好合并技术,为个体个性化提供了一种可扩展、高效的LLM微调替代方案。

英文摘要:

This paper presents a novel approach to aligning large language models (LLMs) with individual human preferences, sometimes referred to as Reinforcement Learning from \textit{Personalized} Human Feedback (RLPHF). Given stated preferences along multiple dimensions, such as helpfulness, conciseness, or humor, the goal is to create an LLM without re-training that best adheres to this specification. Starting from specialized expert LLMs, each trained for one such particular preference dimension, we propose a black-box method that merges their outputs on a per-token level. We train a lightweight Preference Control Model (PCM) that dynamically translates the preference description and current context into next-token prediction weights. By combining the expert models' outputs at the token level, our approach dynamically generates text that optimizes the given preference. Empirical tests show that our method matches or surpasses existing preference merging techniques, providing a scalable, efficient alternative to fine-tuning LLMs for individual personalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑