arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.20548cs.CLcs.AIcs.SI
收藏

PerSo:探究大型语言模型(LLMs)在现实世界说服场景中的社会智能维度

PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

School of Electrical and Computer Engineering, College of Engineering, University of Tehran(德黑兰大学工程学院电气与计算机工程学院);Cardiff University(卡迪夫大学);Institute for Research in Fundamental Sciences (IPM)(基础科学研究所);Tehran Institute for Advanced Studies, Khatam University(哈塔姆大学德黑兰高级研究院)

Ali Zamani, Soroush Esfahanian, Erfan Shahabi, Mohammad Taher Pilehvar, Azadeh Shakery, Yadollah Yaghoobzadeh

AI总结:

该研究推出PerSo基准,利用Reddit真实说服对话评估LLMs的社会智能,发现LLM性能随规模提升,在行为任务上略超人类最佳表现,认知任务仍逊于人类。

摘要:

理解大型语言模型(LLMs)的社会智能对于涉及人机交互和多智能体环境的应用愈发重要。社会智能包含两个方面:认知智能,即推断他人心理状态的能力;行为智能,即基于这些推断选择恰当社会行动以实现社会目标的能力。说服是同时涉及这两个方面的天然测试平台,但现有基准通常使用合成场景孤立地评估它们。我们推出PerSo,一个基于Reddit的ChangeMyView社区真实说服对话评估上述两个方面的基准,采用LLM-in-the-loop流程构建评估任务,并通过人工标注验证。我们对多种LLMs的评估显示,其性能通常随模型规模扩大而提升,但在不同维度存在差异;表现最强的LLM在行为任务上略优于观测到的最佳人类表现,而最佳人类参与者在认知任务上仍领先。

英文摘要:

Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions. The strongest LLM slightly surpasses the best observed human performance on the behavioral task, while the best human participant remains ahead on the cognitive task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑