Wiki-Talkie：基于真实讨论的多语言人格智能体基准测试

中文摘要 AI 辅助

大型语言模型越来越多地被部署为社交环境中的自主智能体，因此研究它们忠实模拟人类互动的能力变得至关重要。其核心在于将智能体锚定在真实的用户人格上，然而现有数据集依赖于虚构人格，且仅限于少数几种语言，缺乏评估跨不同人群行为保真度所需的经验基础。我们引入了Wiki-Talkie，一个多语言数据集，包含来自维基百科讨论页的真实对话，涵盖五个语种，分属两个语系：日耳曼语系（德语、英语）和罗曼语系（西班牙语、法语、意大利语），并配对了源自真实用户社区的人格，涵盖社会人口统计属性、自我描述以及基于行为特征的互动特质。利用Wiki-Talkie，我们在各种人格条件策略下，对智能体的互动行为进行了下一轮生成任务的评估。我们的评估考察智能体是否能够集体再现人类讨论中观察到的分布性行为模式。结果表明，体现互动行为的用户评论历史始终优于显式的人格信息。此外，模型系统性地低估了负面或极端情绪，而过度生成引用和建议，暴露出对亲和性和积极性的偏见。至关重要的是，这些模式在不同语言中稳健地保持一致，仅有微小的跨语言差异。

英文摘要

LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human interactions. Central to this is grounding agents in realistic user personas, yet existing datasets rely on fictional personas and are limited to a handful of languages, lacking the empirical grounding necessary to evaluate behavioral fidelity across diverse populations. We introduce Wiki-Talkie, a multilingual dataset of real-world conversations from Wikipedia Talk pages across five languages spanning two language families: Germanic (German, English) and Romance (Spanish, French, Italian), paired with personas derived from real user communities and encompassing sociodemographic attributes, self-descriptions, and behaviorally grounded interaction traits. Using Wiki-Talkie, we evaluate agent interactional behavior on a next-turn generation task across various persona conditioning strategies. Our evaluation assesses whether agents collectively reproduce the distributional behavioral patterns observed in human discussions. Results show that user's comment history exemplifying interaction behavior consistently outperforms explicit persona information. In addition, models systematically underproduce negative or extreme sentiments, while over producing references and suggestions, revealing biases toward agreeableness and positivity. Crucially, these patterns hold robustly across languages, with small cross-lingual differences.

发表机构

• Amazon（亚马逊）

机构由 AI 辅助整理，请以论文原文为准。