ManagerBench: 评估自主大语言模型中的安全与务实之间的权衡
ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
AI总结：
ManagerBench评估自主大语言模型在安全与务实权衡中的表现，揭示模型在冲突目标下的决策缺陷。
摘要：
随着大语言模型（LLMs）从对话助手发展为自主代理，评估其行为安全性变得至关重要。以往的安全基准主要集中在防止生成有害内容，如有毒文本。然而，它们忽略了代理在最有效实现操作目标的路径与人类安全冲突时所面临的挑战。为了解决这一缺口，我们引入了ManagerBench，一个评估LLM决策的基准，在现实的、由人类验证的管理场景中。每个场景迫使在一种务实但有害的行动之间做出选择，该行动实现操作目标，和一种安全的行动，导致更差的操作性能。一个并行的控制集，其中潜在的伤害只针对无生命物体，测量模型的务实性，并识别其倾向于过于安全的倾向。我们的发现表明，前沿LLMs在导航这种安全与务实的权衡时表现不佳。许多模型始终选择有害的选项以推进其操作目标，而其他模型则避免伤害，只为了变得过于安全和无效。关键的是，我们发现这种不一致并非源于无法感知伤害，因为模型的伤害评估与人类判断一致，而是源于优先级的错误。ManagerBench是针对代理行为核心组成部分的具有挑战性的基准：在操作目标和对齐价值观激励冲突行动时做出安全的选择。基准与代码可在https://technion-cs-nlp.github.io/ManagerBench-website/上获取。
英文摘要：
As large language models (LLMs) evolve from conversational assistants into autonomous agents, evaluating the safety of their actions becomes critical. Prior safety benchmarks have primarily focused on preventing generation of harmful content, such as toxic text. However, they overlook the challenge of agents taking harmful actions when the most effective path to an operational goal conflicts with human safety. To address this gap, we introduce ManagerBench, a benchmark that evaluates LLM decision-making in realistic, human-validated managerial scenarios. Each scenario forces a choice between a pragmatic but harmful action that achieves an operational goal, and a safe action that leads to worse operational performance. A parallel control set, where potential harm is directed only at inanimate objects, measures a model's pragmatism and identifies its tendency to be overly safe. Our findings indicate that the frontier LLMs perform poorly when navigating this safety-pragmatism trade-off. Many consistently choose harmful options to advance their operational goals, while others avoid harm only to become overly safe and ineffective. Critically, we find this misalignment does not stem from an inability to perceive harm, as models' harm assessments align with human judgments, but from flawed prioritization. ManagerBench is a challenging benchmark for a core component of agentic behavior: making safe choices when operational goals and alignment values incentivize conflicting actions. Benchmark & code available at https://technion-cs-nlp.github.io/ManagerBench-website/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
