面向个性化生成的大语言模型鲁棒性基准测试与改进
Benchmarking and Improving LLM Robustness for Personalized Generation
AI总结：
该研究针对大语言模型个性化生成中事实性常被忽视的问题，提出PERG评估框架与PERGData数据集，发现现有模型鲁棒性不足，进而提出Pref-Aligner两阶段方法，平均提升模型鲁棒性25%。
摘要：
近年来，人们对大语言模型（LLM）的响应个性化兴趣日益浓厚。现有评估主要关注响应是否符合用户偏好，但我们认为事实性是同等重要却常被忽视的维度。在个性化语境下，我们将模型定义为鲁棒的：其响应既具备事实准确性，又符合用户偏好。为评估这一特性，我们提出PERG（一种可扩展的LLM鲁棒性评估框架），以及新数据集PERGData。我们采用不同提示方法，评估了来自5个模型家族的14个模型。研究结果表明，当前LLM在鲁棒个性化方面存在困难：即便是性能最强的模型（GPT-4.1、LLaMA3-70B），在无个性化时原本成功的案例中，有5%无法维持正确性；而更小的模型（如7B规模）失败率可超过20%。进一步分析显示，查询性质和用户偏好类型会显著影响鲁棒性。为缓解这些失败问题，我们提出Pref-Aligner，这一两阶段方法可使各模型的鲁棒性平均提升25%。我们的工作揭示了当前评估实践中的关键缺口，并引入了工具和指标以支持更可靠、符合用户需求的LLM部署。
英文摘要：
Recent years have witnessed a growing interest in personalizing the responses of large language models (LLMs). While existing evaluations primarily focus on whether a response aligns with a user's preferences, we argue that factuality is an equally important yet often overlooked dimension. In the context of personalization, we define a model as robust if its responses are both factually accurate and align with the user preferences. To assess this, we introduce PERG, a scalable framework for evaluating robustness in LLMs, along with a new dataset, PERGData. We evaluate fourteen models from five different model families using different prompting methods. Our findings show that current LLMs struggle with robust personalization: even the strongest models (GPT-4.1, LLaMA3-70B) fail to maintain correctness in 5% of previously successful cases without personalization, while smaller models (e.g., 7B-scale) can fail more than 20% of the time. Further analysis reveals that robustness is significantly affected by the nature of the query and the type of user preference. To mitigate these failures, we propose Pref-Aligner, a two-stage approach that improves robustness by an average of 25% across models. Our work highlights critical gaps in current evaluation practices and introduces tools and metrics to support more reliable, user-aligned LLM deployments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
