他们是恋人还是朋友？评估LLMs在英语和韩语对话中的社交推理能力
Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
AI总结：
本文通过SCRIPTS数据集评估LLMs在英语和韩语对话中推断人物关系的能力，发现当前模型在英语和韩语中的准确率分别为75-80%和58-69%，且存在显著的社交推理局限性。
摘要：
随着LLMs在现实世界中的应用增加，其在人际沟通中的社交推理能力变得至关重要。为了探索其能力，我们引入了包含1100个对话的SCRIPTS数据集，该数据集源自电影脚本，并提出了一项基于SCRIPTS的任务，用于评估LLMs推断对话中说话者之间社会关系（如朋友、恋人）的能力。在我们的任务上评估九个模型，当前LLMs在英语数据集上达到约75-80%的准确率，在韩语中为58-69%。模型在两种语言中预测不寻常关系的比例为10-25%。此外，我们发现思考模型和链式推理提示对社交推理提供最小的益处，有时会放大社交偏见。总之，当前LLMs的社交推理能力存在显著局限，尤其是在韩语中，这突显了开发跨语言的社会感知LLMs的必要性。
英文摘要：
As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SCRIPTS, a 1.1k-dialogue dataset in English and Korean, sourced from movie scripts and propose a social reasoning task based on SCRIPTS that evaluates the capacity of LLMs to infer the social relationships (e.g., friends, lovers) between speakers in each dialogue. Evaluating nine models on our task, current LLMs achieve around 75--80% on the English dataset and 58--69% in Korean, and models predict an Unlikely relationship in 10--25% of responses in both languages. Furthermore, we find that thinking models and chain-of-thought prompting provide minimal benefits for social reasoning and occasionally amplify social biases. In sum, there are significant limitations in current LLMs' social reasoning capabilities, especially for Korean, highlighting the need for efforts to develop socially-aware LLMs across languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
