识别脱离角色行为：开放式生成中人设保真度的原子级评估
Spotting Out-of-Character Behavior: Atomic-Level Evaluation of Persona Fidelity in Open-Ended Generation
AI总结：
针对LLMs在开放式生成中偏离指定人设的脱离角色（OOC）行为，提出一种原子级评估框架，通过三个关键指标在更细粒度上量化人设对齐与一致性，有效检测传统方法忽略的细微人设偏差。
摘要：
确保大型语言模型（LLMs）的人设保真度对于维持连贯且引人入胜的人机交互至关重要。然而，LLMs经常表现出脱离角色（OOC）行为，即生成的回复偏离了指定的人设，导致影响模型可靠性的不一致性。现有的评估方法通常对整个回复给出单一分数，难以捕捉细微的人设偏差，尤其是在长文本生成中。为了解决这一局限性，我们提出了一种原子级评估框架，以更细的粒度量化人设保真度。我们的三个关键指标衡量了生成内部和跨生成的人设对齐与一致性程度。通过识别真实用户可能遇到的细微偏差，我们的方法能够更精确、更真实地评估人设保真度。通过实验，我们证明了我们的框架能有效检测出先前方法忽略的人设不一致性。通过分析不同任务和性格类型下的人设保真度，我们揭示了任务结构和人设期望如何影响模型的适应性，突显了在保持一致人设表达方面面临的挑战。
英文摘要：
Ensuring persona fidelity in large language models (LLMs) is essential for maintaining coherent and engaging human-AI interactions. However, LLMs often exhibit Out-of-Character (OOC) behavior, where generated responses deviate from an assigned persona, leading to inconsistencies that affect model reliability. Existing evaluation methods typically assign single scores to entire responses, struggling to capture subtle persona misalignment, particularly in long-form text generation. To address this limitation, we propose an atomic-level evaluation framework that quantifies persona fidelity at a finer granularity. Our three key metrics measure the degree of persona alignment and consistency within and across generations. Our approach enables a more precise and realistic assessment of persona fidelity by identifying subtle deviations that real users would encounter. Through our experiments, we demonstrate that our framework effectively detects persona inconsistencies that prior methods overlook. By analyzing persona fidelity across diverse tasks and personality types, we reveal how task structure and persona desirability influence model adaptability, highlighting challenges in maintaining consistent persona expression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
