RLPF：面向大语言模型用户摘要的预测反馈强化学习
RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs
AI总结：
针对预训练LLM生成的用户摘要简洁但缺乏下游任务上下文、难以利用长噪声用户历史的问题，提出RLPF方法微调LLM生成适配下游任务的简洁用户摘要，显著提升下游性能与摘要质量，兼具泛化性。
摘要：
基于大语言模型（LLM）的个性化智能体系统利用大语言模型从用户过往活动中预测其行为。然而，这类系统的有效性往往取决于能否有效利用海量、冗长的用户历史数据，因为此类数据本身存在噪声且篇幅过长。现有预训练LLM生成的摘要可能简洁，但缺乏下游任务所需的上下文，削弱了其在个性化系统中的实用性。为应对这些挑战，我们提出了预测反馈强化学习（Reinforcement Learning from Prediction Feedback，RLPF）。RLPF对LLM进行微调，以生成简洁、人类可读的用户摘要，且针对下游任务性能进行了优化。通过最大化生成摘要的有用性，RLPF有效提炼了海量用户历史数据，同时保留了下游任务所需的关键信息。我们的实证评估显示，该方法在下游任务外在效用和摘要内在质量两方面均取得显著提升：下游任务性能较基线方法最高提升22%，在事实性、抽象性和可读性方面的胜率最高达84.59%。RLPF还实现了高达74%的上下文长度缩减，同时在19个未见过的任务和/或数据集中的16个上提升了性能，展现出良好的泛化能力。该方法通过将冗长、含噪的用户历史有效转化为信息丰富且人类可读的表征，为增强LLM个性化效果提供了一种有前景的解决方案。
英文摘要：
LLM-powered personalization agent systems employ Large Language Models (LLMs) to predict users' behavior from their past activities. However, their effectiveness often hinges on the ability to effectively leverage extensive, long user historical data due to its inherent noise and length of such data. Existing pretrained LLMs may generate summaries that are concise but lack the necessary context for downstream tasks, hindering their utility in personalization systems. To address these challenges, we introduce Reinforcement Learning from Prediction Feedback (RLPF). RLPF fine-tunes LLMs to generate concise, human-readable user summaries that are optimized for downstream task performance. By maximizing the usefulness of the generated summaries, RLPF effectively distills extensive user history data while preserving essential information for downstream tasks. Our empirical evaluation demonstrates significant improvements in both extrinsic downstream task utility and intrinsic summary quality, surpassing baseline methods by up to 22% on downstream task performance and achieving an up to 84.59% win rate on Factuality, Abstractiveness, and Readability. RLPF also achieves a remarkable 74% reduction in context length while improving performance on 16 out of 19 unseen tasks and/or datasets, showcasing its generalizability. This approach offers a promising solution for enhancing LLM personalization by effectively transforming long, noisy user histories into informative and human-readable representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
