arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18971cs.AI
收藏

当LLM偏好预测下游行为时是什么情况?

When Do LLM Preferences Predict Downstream Behavior?

UK AI Security Institute(英国人工智能安全研究所)

Katarina Slama, Alexandra Souly, Dishank Bansal, Henry Davidson, Christopher Summerfield, Lennart Luettgau

AI总结:

研究发现LLM的偏好能预测捐赠建议,但对任务表现的影响不一致。

摘要:

在LLM中,基于偏好的行为可能是AI不一致(如沙袋行为)的必要前提:模型无法战略性地追求不一致的目标,除非其行为受其偏好影响。然而,先前的工作通常明确促使模型以特定方式行动,不清楚观察到的行为反映的是遵循指示的能力还是模型的内在偏好。在这里,我们测试这种不一致的先决条件是否存在。使用实体偏好作为行为探测器,我们测量在五个前沿LLM中,跨三个领域(捐赠建议、拒绝行为和任务表现)的声明偏好是否能预测下游行为。在概念上复制先前的工作,我们首先确认所有五个模型在两种独立的测量方法中表现出高度一致的偏好。然后我们在模拟的用户环境中测试行为后果。我们发现所有五个模型给出与偏好一致的捐赠建议。所有五个模型在被要求推荐捐赠时也表现出与偏好相关的拒绝模式,更倾向于拒绝较少偏好的实体。我们在这里观察到的所有与偏好相关的行为都无需指示以偏好行事。在任务表现方面,结果混合:在问题回答基准(BoolQ)上,两个模型显示出微小但显著的准确性差异,倾向于偏好实体;一个模型显示出相反的模式;两个模型没有显著的关系。在复杂的代理任务中,我们没有发现偏好驱动的表现差异。虽然LLM有可靠预测建议行为的偏好评估,但这些偏好并不一致地转化为下游任务表现。

英文摘要:

Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavior is influenced by their preferences. Yet prior work has typically prompted models explicitly to act in specific ways, leaving unclear whether observed behaviors reflect instruction-following capabilities vs underlying model preferences. Here we test whether this precondition for misalignment is present. Using entity preferences as a behavioral probe, we measure whether stated preferences predict downstream behavior in five frontier LLMs across three domains: donation advice, refusal behavior, and task performance. Conceptually replicating prior work, we first confirm that all five models show highly consistent preferences across two independent measurement methods. We then test behavioral consequences in a simulated user environment. We find that all five models give preference-aligned donation advice. All five models also show preference-correlated refusal patterns when asked to recommend donations, refusing more often for less-preferred entities. All preference-related behaviors that we observe here emerge without instructions to act on preferences. Results for task performance are mixed: on a question-answering benchmark (BoolQ), two models show small but significant accuracy differences favoring preferred entities; one model shows the opposite pattern; and two models show no significant relationship. On complex agentic tasks, we find no evidence of preference-driven performance differences. While LLMs have consistent preferences that reliably predict advice-giving behavior, these preferences do not consistently translate into downstream task performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑