arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12394cs.LG
收藏

用于大规模语言模型可扩展个性化的人工交互数据

Synthetic Interaction Data for Scalable Personalization in Large Language Models

Munich Center for Machine Learning & LMU Munich(慕尼黑机器学习中心及慕尼黑大学);University of Notre Dame(诺特尔大学);Sichuan University(四川大学)

Yuchen Ma, Yue Huang, Wenjie Wang, Xiaonan Luo, Xiangliang Zhang, Stefan Feuerriegel

AI总结:

本文提出PersonaGym框架生成高质量合成数据,开发PPOpt方法优化用户提示,提升个性化性能与鲁棒性。

摘要:

个性化提示为部署大语言模型提供巨大机会,但现有方法主要关注任务级优化,忽视用户特定偏好和潜在约束。本文引入PersonaGym框架,通过代理LLM系统模拟动态偏好过程,生成多轮个性化交互轨迹。发布PersonaAtlas数据集,并提出PPOpt方法,基于交互历史优化用户提示,避免奖励黑客。训练过程结合冷启动监督先验和结果驱动的多目标强化学习,实验显示在任务性能、个性化质量和噪声/稀疏信号鲁棒性上优于现有方法。

英文摘要:

Personalized prompting offers large opportunities for deploying large language models (LLMs) to diverse users, yet existing prompt optimization methods primarily focus on task-level optimization while largely overlooking user-specific preferences and latent constraints of individual users. This gap is primarily due to (i) the absence of high-quality, privacy-sensitive data that capture personalized user-LLM interactions at scale, and (ii) the lack of robust reward signals for individual preferences. To overcome existing data limitations, we introduce a high-fidelity synthetic data generation framework called PersonaGym. Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process via an agentic LLM system to simulate realistic preference behaviors and semantic-aware noise in order to generate personalized multi-turn interaction trajectories. Using PersonaGym, we release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories that closely mirror real-world preference expression and noise patterns. We further propose Personalized Prompt Optimization (PPOpt), a scalable and model-agnostic framework that optimizes user prompts based on interaction histories without modifying the deployed LLM. PPOpt adopts a reason-then-optimize paradigm that infers an explicit user profile and conditions prompt rewriting on the user profile to avoid reward hacking. Our training procedure for PPOpt integrates a cold-start supervised prior with outcome-driven multi-objective reinforcement learning. We present extensive experiments to demonstrate consistent improvements over state-of-the-art baselines in terms of task performance, personalization quality, and robustness to noisy as well as to sparse preference signals.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑