反思式个性化优化：面向黑盒大语言模型的事后重写框架
Reflective Personalization Optimization: A Post-hoc Rewriting Framework for Black-Box Large Language Models
AI总结：
本文提出反思式个性化优化（RPO）框架，通过将内容生成与用户偏好对齐解耦，先由基础模型生成通用响应，再由外部反思模块经监督微调和强化学习进行事后重写，在 LaMP 基准上显著超越现有方法，实现了模型无关的高效个性化。
摘要：
黑盒大语言模型（LLMs）的个性化是一项关键但具有挑战性的任务。现有方法主要依赖上下文注入，即将用户历史嵌入提示中以直接引导生成过程。然而，这种单步范式给模型带来了双重负担：既要生成准确内容，又要同时符合用户特定风格。这往往导致一种折中，损害输出质量并限制精确控制。为应对这一根本性张力，我们提出反思式个性化优化（Reflective Personalization Optimization，RPO），这是一个新颖的框架，通过将内容生成与对齐解耦来重新定义个性化范式。RPO 在两个不同阶段运行：首先，基础模型生成高质量、通用的响应；然后，外部反思模块显式地重写该输出以符合用户偏好。该反思模块采用两阶段过程进行训练。首先，在结构化重写轨迹上使用监督微调，以建立核心的个性化推理策略，建模从通用响应到用户对齐响应的转变。随后，应用强化学习进一步优化和提升个性化输出的质量。在 LaMP 基准上的全面实验表明，RPO 通过将内容生成与个性化解耦，显著优于最先进的基线方法。这些发现强调了显式响应塑造相对于隐式上下文注入的优越性。此外，RPO 引入了一个高效、模型无关的个性化层，可以无缝集成到任何底层基础模型中，为用户中心生成场景开辟了一个新的有效方向。
英文摘要：
The personalization of black-box large language models (LLMs) is a critical yet challenging task. Existing approaches predominantly rely on context injection, where user history is embedded into the prompt to directly guide the generation process. However, this single-step paradigm imposes a dual burden on the model: generating accurate content while simultaneously aligning with user-specific styles. This often results in a trade-off that compromises output quality and limits precise control. To address this fundamental tension, we propose Reflective Personalization Optimization (RPO), a novel framework that redefines the personalization paradigm by decoupling content generation from alignment. RPO operates in two distinct stages: first, a base model generates a high-quality, generic response; then, an external reflection module explicitly rewrites this output to align with the user's preferences. This reflection module is trained using a two-stage process. Initially, supervised fine-tuning is employed on structured rewriting trajectories to establish a core personalized reasoning policy that models the transformation from generic to user-aligned responses. Subsequently, reinforcement learning is applied to further refine and enhance the quality of the personalized outputs. Comprehensive experiments on the LaMP benchmark demonstrate that RPO, by decoupling content generation from personalization, significantly outperforms state-of-the-art baselines. These findings underscore the superiority of explicit response shaping over implicit context injection. Moreover, RPO introduces an efficient, model-agnostic personalization layer that can be seamlessly integrated with any underlying base model, paving the way for a new and effective direction in user-centric generation scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
