PREF：LLM中个性化文本生成的无参考评估
PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs
AI总结：
提出PREF框架，通过覆盖、偏好和评分三步流程，在无需参考答案的情况下联合评估输出质量和用户对齐，显著提升了个性化文本生成的评估准确性和人类对齐度。
摘要：
个性化文本生成对于以用户为中心的信息系统至关重要，但大多数评估方法忽略了用户的个体性。我们引入了PREF，一个个性化无参考评估框架，它无需黄金个性化参考即可联合衡量通用输出质量和用户特定对齐。PREF在一个三步流程中运行：（1）覆盖阶段使用大型语言模型（LLM）生成涵盖事实性、连贯性和完整性等通用标准的全面、特定于查询的指南；（2）偏好阶段使用目标用户的个人资料、明示或推断的偏好以及上下文，对这些因素进行重新排序和选择性增强，生成个性化评估准则；（3）评分阶段应用LLM裁判根据此准则对候选答案进行评分，确保基线充分性的同时捕捉主观优先级。这种覆盖与偏好的分离提高了鲁棒性、透明度和可重用性，并允许较小的模型近似较大模型的个性化质量。在PrefEval基准上的实验（包括隐式偏好遵循任务）表明，与强基线相比，PREF实现了更高的准确率、更好的校准以及与人类判断更紧密的对齐。通过实现可扩展、可解释和用户对齐的评估，PREF为更可靠的个性化语言生成系统的评估和开发奠定了基础。
英文摘要：
Personalised text generation is essential for user-centric information systems, yet most evaluation methods overlook the individuality of users. We introduce \textbf{PREF}, a \textbf{P}ersonalised \textbf{R}eference-free \textbf{E}valuation \textbf{F}ramework that jointly measures general output quality and user-specific alignment without requiring gold personalised references. PREF operates in a three-step pipeline: (1) a coverage stage uses a large language model (LLM) to generate a comprehensive, query-specific guideline covering universal criteria such as factuality, coherence, and completeness; (2) a preference stage re-ranks and selectively augments these factors using the target user's profile, stated or inferred preferences, and context, producing a personalised evaluation rubric; and (3) a scoring stage applies an LLM judge to rate candidate answers against this rubric, ensuring baseline adequacy while capturing subjective priorities. This separation of coverage from preference improves robustness, transparency, and reusability, and allows smaller models to approximate the personalised quality of larger ones. Experiments on the PrefEval benchmark, including implicit preference-following tasks, show that PREF achieves higher accuracy, better calibration, and closer alignment with human judgments than strong baselines. By enabling scalable, interpretable, and user-aligned evaluation, PREF lays the groundwork for more reliable assessment and development of personalised language generation systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
