RoleConflictBench: 一个用于评估LLM上下文敏感性的角色冲突场景基准
RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity
AI总结：
本文提出RoleConflictBench基准，通过生成13000个角色冲突场景，评估LLM在角色冲突中的上下文敏感性，发现模型更倾向于遵循角色偏好而非动态上下文线索。
摘要：
人们常面临角色冲突——多重角色期望碰撞导致无法同时满足的社会困境。随着大语言模型（LLM）日益参与这些社会动态，一个关键问题浮现：当面临此类困境时，LLM更倾向于优先考虑动态上下文线索还是已学习的偏好？为此，我们引入RoleConflictBench，一个新型基准，用于测量LLM在角色冲突场景中的上下文敏感性。为在主观领域实现客观评估，我们采用情境紧迫性作为决策约束。通过三阶段流程，我们生成了超过13000个真实场景，涵盖65种角色和五个社会领域，通过系统性地变化竞争情境的紧迫性。这种受控设置使我们能够定量测量上下文敏感性，确定模型决策是否符合情境上下文或被已学习的角色偏好所覆盖。对10个LLM的分析显示，模型显著偏离此客观基线。其决策主要由对特定社会角色的偏好所主导。
英文摘要：
People often encounter role conflicts -- social dilemmas where the expectations of multiple roles clash and cannot be simultaneously fulfilled. As large language models (LLMs) increasingly navigate these social dynamics, a critical research question emerges. When faced with such dilemmas, do LLMs prioritize dynamic contextual cues or the learned preferences? To address this, we introduce RoleConflictBench, a novel benchmark designed to measure the contextual sensitivity of LLMs in role conflict scenarios. To enable objective evaluation within this subjective domain, we employ situational urgency as a constraint for decision-making. We construct the dataset through a three-stage pipeline that generates over 13,000 realistic scenarios across 65 roles in five social domains by systematically varying the urgency of competing situations. This controlled setup enables us to quantitatively measure contextual sensitivity, determining whether model decisions align with the situational contexts or are overridden by the learned role preferences. Our analysis of 10 LLMs reveals that models substantially deviate from this objective baseline. Instead of responding to dynamic contextual cues, their decisions are predominantly governed by the preferences toward specific social roles.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
