个性化轮级用户对话满意度基准
Personalized Turn-Level User Conversation Satisfaction Benchmark
AI总结：
针对AI助手响应的个性化满意度评估问题，提出结合用户记忆与目标轮上下文的满意度评估器，并构建PersTurnBench基准，通过回放实现生成模型的受控比较。
摘要：
用户对AI助手的满意度高度个性化：同一响应可能满足一个用户但令另一个失望，取决于每个用户的期望以及他们之前询问的内容。现有的自动评估方法大多衡量通用响应质量，难以判断某个响应在特定轮次是否满足用户。我们将此问题作为个性化轮级用户对话满意度评估进行研究。我们构建了一个对话满意度评估器，将紧凑的用户记忆与目标轮上下文相结合，生成满意度分数和不满意的理由。与人类满意度标注的元评估表明，个性化记忆和事后分数校准在有序一致性和不满意轮次检测上优于监督式、检索式和通用LLM作为评判者的基线。我们进一步引入了PersTurnBench，这是一个个性化轮级用户对话满意度基准，通过回放使用经过验证的评估器来评估生成模型。通过固定回放状态，PersTurnBench能够在无需为每个候选模型收集新人工标签的情况下，对通用生成模型和记忆增强的个性化系统进行受控比较。该评估器和基准让研究人员能够在无需为每个模型收集新用户反馈的情况下，比较候选生成模型在个性化满意度上的表现。
英文摘要：
User satisfaction with AI assistants is highly personalized: the same response may satisfy one user but disappoint another depending on what each user expects and what they have asked for before. Existing automatic evaluation methods mostly measure generic response quality, making it difficult to judge whether a response satisfies a user at a specific turn. We study this problem as personalized turn-level user conversation satisfaction evaluation. We build a conversation satisfaction evaluator that combines compact user memories with target-turn context to produce satisfaction scores and dissatisfaction-oriented rationales. Meta-evaluation against human satisfaction annotations shows that personalized memory and post-hoc score calibration improve ordinal agreement and dissatisfied-turn detection over supervised, retrieval-based, and generic LLM-as-a-judge baselines. We further introduce PersTurnBench, a personalized turn-level user conversation satisfaction benchmark that uses the verified evaluator to assess generation models via replay. By holding the replay state fixed, PersTurnBench enables controlled comparison of generic generation models and memory-augmented personalized systems without new human labels for every candidate model. The evaluator and benchmark let researchers compare candidate generation models on personalized satisfaction without collecting new user feedback for every model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
