PAD：解码时大语言模型的个性化对齐
PAD: Personalized Alignment of LLMs at Decoding-Time
AI总结：
本文提出无需训练的解码时个性化对齐框架 PAD，通过 token 级个性化奖励引导 LLM 输出，在多样、未见偏好及跨模型场景中优于训练式方法。
摘要：
与个性化偏好保持一致是一项重大挑战，因为这些偏好会因文化、教育和政治差异而显著不同，而传统对齐方法又需要高昂的计算成本和大量数据。为此，本文提出 Personalized Alignment at Decoding-time（PAD，解码时个性化对齐），这是一种新颖框架，旨在推理阶段使 LLM 输出与多样化的个性化偏好保持一致，无需额外训练。通过引入独特的个性化奖励建模策略，该框架将文本生成过程与个性化偏好解耦，便于生成可泛化的 token 级个性化奖励。PAD 算法利用这些奖励引导解码过程，根据个性化偏好动态调整基础模型的预测。大量实验结果表明，PAD 不仅在与多样化偏好对齐方面优于现有基于训练的对齐方法，而且对训练期间未见过的偏好表现出显著的泛化能力，并具备跨不同基础模型的可扩展性。本工作提升了 LLM 在实时应用中满足用户需求的能力，代表个性化 LLM 对齐迈出的重要一步。
英文摘要：
Aligning with personalized preferences, which vary significantly across cultural, educational, and political differences, poses a significant challenge due to the computational costs and data demands of traditional alignment methods. In response, this paper presents Personalized Alignment at Decoding-time (PAD), a novel framework designed to align LLM outputs with diverse personalized preferences during the inference phase, eliminating the need for additional training. By introducing a unique personalized reward modeling strategy, this framework decouples the text generation process from personalized preferences, facilitating the generation of generalizable token-level personalized rewards. The PAD algorithm leverages these rewards to guide the decoding process, dynamically tailoring the base model's predictions to personalized preferences. Extensive experimental results demonstrate that PAD not only outperforms existing training-based alignment methods in terms of aligning with diverse preferences but also shows significant generalizability to preferences unseen during training and scalability across different base models. This work advances the capability of LLMs to meet user needs in real-time applications, presenting a substantial step forward in personalized LLM alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
