HyPerAlign：通过假设生成实现可解释的个性化大语言模型对齐
HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
AI总结：
针对现有LLM对齐仅适配平均用户偏好的问题，提出可解释、样本高效的假设驱动个性化方法HyPerAlign，通过推断用户特征假设生成定制输出，在两项个性化任务上性能显著优于偏好微调方法。
摘要：
对齐算法被广泛用于基于偏好标注将大语言模型（LLM）与人类用户对齐。通常情况下，这些（往往存在分歧的）偏好会在多样化的用户群体中进行聚合，最终得到的微调模型是与“平均用户”偏好对齐的。然而，当前模型由个体用户在高度特定的上下文与场景中使用，这凸显了实现依赖于用户的偏好控制的必要性。在本研究中，我们解决了让LLM输出面向用户个性化的问题。我们的目标是生成适配特定个体的定制化回复，而非模拟多样化群体集体声音的通用输出。我们提出了HyPerAlign，这是一种面向LLM的、可解释且样本高效的假设驱动型个性化方法。给定某个特定用户撰写的少样本示例，我们首先推断关于其沟通策略、性格与写作风格的假设，随后将这些假设与用户特定属性作为提示输入LLM，以生成定制化输出。我们在两项不同的个性化任务上开展了实验，分别是作者归属与协商对齐，所使用的数据集来自多个不同领域（新闻文章、博客帖子、电子邮件、越狱基准）。结果表明，与基于偏好的微调方法相比，假设驱动的LLM个性化方法更具优势。在作者归属任务中，在多样化的用户画像与LLM模型上，HyPerAlign生成的内容相较于当前最优的偏好微调方法始终保持很高的胜率（通常>90%）。在协商对齐任务中，LLM模型的有用性平均提升了最高达70%。总体而言，HyPerAlign为实现LLM面向个体用户的个性化提供了一种可解释且样本高效的策略。
英文摘要：
Alignment algorithms are widely used to align large language models (LLMs) to human users based on preference annotations. Typically these (often divergent) preferences are aggregated over a diverse set of users, resulting in fine-tuned models that are aligned to the ``average-user'' preference. Nevertheless, current models are used by individual users in very specific contexts and situations, emphasizing the need for user-dependent preference control. In this work we address the problem of personalizing LLM outputs to their users. We aim to generate customized responses tailored to specific individuals instead of generic outputs that emulate the collective voices of diverse populations. We propose HyPerAlign, an interpretable and sample-efficient hypothesis-driven personalization approach for LLM models. Given few-shot examples written by a particular user, we first infer hypotheses about their communication strategies, personality, and writing style, then prompt LLM models with these hypotheses and user-specific attributes to generate customized outputs. We conduct experiments on two different personalization tasks, namely authorship attribution and deliberative alignment, with datasets from diverse domains (news articles, blog posts, emails, jailbreaking benchmarks). Results demonstrate the superiority of hypothesis-driven LLM personalization compared to preference-based fine-tuning methods. For authorship attribution, HyPerAlign generations have consistently high win-rates (commonly $> 90\%$) against state-of-the-art preference fine-tuning approaches across diverse user profiles and LLM models. For deliberative alignment, the helpfulness of LLM models is improved by up to $70\%$ on average. Overall, HyPerAlign represents an interpretable and sample-efficient strategy for the personalization of LLM models to individual users.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
