DyMT-ESB：用户与LLM交互中社会偏见的动态多轮评估
DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions
浏览论文内容
中文总结 AI 辅助
本文提出DyMT-ESB协议，通过响应条件生成后续查询和可变轮数，评估LLM多轮交互中的社会偏见动态，发现后期出现、非单调及重新出现的偏见模式，强调轮级动态分析的重要性。
中文摘要 AI 辅助
警告：本文包含刻板印象和社会偏见的示例。LLM越来越多地被公众用于交互式场景，因此评估模型在多轮对话场景中的行为对于安全性（包括与刻板印象相关的危害）至关重要。然而，现有的多轮社会偏见评估通常依赖于预先指定或基于模板的用户输入，这些输入不会根据模型响应进行调整，并且通常事先假设固定的对话长度。在本文中，我们使用一种受控评估协议研究响应条件下的多轮交互中的社会偏见动态，该协议根据不断演变的对话历史生成后续用户查询，并允许在可变轮数上进行评估。实验结果表明，LLM即使在连贯的、响应条件下的多轮交互中也会表现出社会偏见，揭示了后期出现的偏见、非单调的偏见模式以及偏见的重新出现。这些结果促使评估超越固定轮次、预先脚本化的协议。我们的发现强调了将社会偏见作为轮级动态现象进行分析的重要性。
英文摘要
Warning: This paper contains examples of stereotypes and social bias. LLMs are increasingly used in interactive settings by the general public, making the evaluation of model behavior in multi-turn conversational scenarios important for safety, including stereotyping-related harms. However, existing multi-turn social bias evaluations often rely on pre-specified or template-based user inputs that do not adapt to model responses and typically assume a fixed dialogue length in advance. In this paper, we study social bias dynamics in response-conditioned multi-turn interactions using a controlled evaluation protocol that generates follow-up user queries from the evolving dialogue history and allows evaluation over variable numbers of turns. Experimental results show that LLMs exhibit social bias even in coherent, response-conditioned multi-turn interactions, revealing late-emerging bias, non-monotonic bias patterns, and bias re-emergence. These results motivate evaluations that extend beyond fixed-turn, pre-scripted protocols. Our findings highlight the importance of analyzing social bias as a turn-level dynamic phenomenon.
发表机构
- Institute of Science Tokyo（东京科学大学）
- MBZUAI（穆罕默德·本·扎耶德人工智能大学）
- Third Intelligence
- The University of Liverpool（利物浦大学）
机构由 AI 辅助整理，请以论文原文为准。
