他内向吗?识别缺失的相关人设以实现忠实的用户模拟
Is He Extroverted? Identifying Missing Relevant Personas for Faithful User Simulation
AI总结:
本文研究了识别缺失人设维度以提升用户模拟的准确性,提出了PICQ-drama基准测试集,并通过评估揭示了LLM与人类在认知上的差异。
摘要:
现有用户模拟方法专注于生成对话中用户般的回应,通常假设提供的个性足够生成此类回应,而未验证关键个性是否被提供。为此,本文研究了识别在模拟用户回复时缺失的相关人设维度(如
英文摘要:
Existing user simulation approaches focus on generating user-like responses in dialogue. They often assume that the provided persona is sufficient for producing such responses, without verifying whether critical personas are supplied. This raises concerns about the validity of simulation results. To address this issue, we study the task of identifying persona dimensions (e.g., "whether the user is price-sensitive") that are relevant but missing in simulating a user's reply for a given dialogue context. We introduce PICQ-drama (constructed from TVShowGuess), a benchmark of context-aware choice questions, annotated with missing persona dimensions whose absence leads to ambiguous user choices. We further design diverse evaluation criteria for missing persona identification. Benchmarking leading LLMs on our PICQ-drama dataset demonstrates the feasibility of this task. Evaluation across diverse criteria, along with further analyses, reveals cognitive differences between LLMs and humans and highlights the distinct roles of different persona categories in shaping responses.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。