评估大语言模型中行为倾向的对齐情况
Evaluating Alignment of Behavioral Dispositions in LLMs
AI总结：
本研究通过情境判断测试评估大语言模型行为倾向与人类倾向的对齐情况，发现模型在不同共识场景下表现出过度自信、偏离共识及跨模型特质模式，揭示了LLMs声明价值与实际行为之间的差距。
摘要：
随着大语言模型（LLMs）逐渐融入我们的日常生活，理解其行为变得至关重要。在本研究中，我们关注行为倾向——这些内在倾向塑造社交情境中响应的底层倾向——并引入了一个框架来研究LLMs所表达的倾向与人类倾向之间的对齐程度。我们的方法基于已建立的心理学问卷，但通过将人类自我报告陈述转化为情境判断测试（SJTs）来适应LLMs。这些SJTs通过在现实用户-助手场景中提出自然的建议来评估行为。我们生成了2500个SJTs，每个都经过三个人类标注者验证，并从550名参与者的大池中收集了每个SJTs的10名标注者的首选行动。在涉及25个LLMs的全面研究中，我们发现模型往往不反映人类偏好的分布：（1）在人类共识较低的场景中，LLMs在单一响应上表现出过度自信；（2）当人类共识较高时，较小的模型显著偏离，甚至一些前沿模型在15-20%的情况下不反映共识；（3）特质可以表现出跨LLM模式，例如在人类共识倾向于克制的情境中，LLMs可能鼓励情绪表达。最后，将心理测量陈述直接映射到行为场景为评估自我报告的预测效度提供了独特机会，揭示了LLMs声明的价值与揭示行为之间存在显著差距。
英文摘要：
As people turn to LLMs for social advice, understanding their behavior in such contexts becomes essential. In this work, we focus on behavioral dispositions: the underlying tendencies that shape responses in social contexts. We introduce STAR, a framework for studying how closely the dispositions expressed by LLMs align with those of humans. STAR builds on established psychological questionnaires, adapting their items into realistic advice-seeking scenarios, as self-report may not transfer to actual advisory behavior. Using STAR, we construct a dataset of 23k scenarios, each validated by 3 raters and annotated with preferences from 10 participants. Across 25 LLMs, we find that (1) when human consensus is high, frontier models can fail to reflect it in 15-20% of cases, and smaller models fail at substantially higher rates; (2) when humans disagree, LLM recommendations are substantially less diverse than human choices, both within individual models and even across models from different providers, potentially narrowing the range of options users are guided toward; (3) LLMs' self-reported values are poor predictors of their recommendations. To support future research we make our dataset and code publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
