arXiv 2510.143512025-10-18cs.CLcs.AI

超越单一世界：跨多元宇宙情境的超级英雄角色扮演基准评测

Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts

Faculty of Engineering, Thammasat School of Engineering, Thammasat University（工程学院，泰国朱拉安高校工程学院，泰国朱拉安大学）；Department of Computer Engineering and Digital Technology, Faculty of Engineering, Chulalongkorn University（计算机工程与数字技术系，工程学院，朱拉安大学）；Artificial Intelligence Association of Thailand（泰国人工智能协会）；School of Biomedical Engineering & Imaging Sciences, King’s College London（生物医学工程与成像科学学院，伦敦国王学院）；Siriraj Informatics and Data Innovation Center (SIData+), Faculty of Medicine, Siriraj Hospital, Mahidol University（Siriraj信息与数据创新中心（SIData+），医学学院，Siriraj医院，玛希道大学）

Perapard Ngokpol, Kun Kerdthaisong, Pasin Buakhaw, Pitikorn Khlaisamniang, Supasate Vorathammathorn, Piyalitt Ittichaiwong, Nutchanon Yongsatianchot

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Beyond One World基准，通过30位英雄、90个正典版本及正典事件与道德困境任务，评估LLM跨多元宇宙角色一致性，并以Think-Act Matching衡量推理与行动对齐。

摘要：

大语言模型（LLM）正越来越多地被用作角色扮演智能体，但它们忠实且一致地刻画特定版本角色的能力——例如漫画宇宙和电影宇宙中的超级英雄——仍未得到充分研究。Marvel和DC等超级英雄正典提供了丰富的测试平台：数十年的叙事使同一角色产生了多个化身，它们拥有不同的经历、价值观和道德准则。为研究这一问题，我们提出Beyond One World，这是一个基于角色 grounding 的角色扮演基准，涵盖30位标志性英雄和90个正典特定版本。该基准包含两项任务：（i）Canon Events，考察模型对关键人生阶段的事实回忆；（ii）Moral Dilemmas，让模型面对具有伦理争议的情境。我们在一个将内部审慎（“思考”）与外在决策（“行动”）分离的框架下，对回答的正典准确性和推理保真度进行评分。我们进一步提出Think-Act Matching指标，用于量化理由与行动之间的一致性，并作为模型可信度的代理指标。在面向推理和非面向推理模型上的实验得出三项发现：（1）思维链提示可提升较弱模型的叙事连贯性，但会降低较强模型的正典准确性；（2）同一角色不同版本之间的跨版本泛化仍是主要障碍；（3）模型通常只擅长思考或行动中的一项，很少两者兼优。Beyond One World揭示了多元宇宙一致性和推理对齐方面的关键差距，为角色扮演LLM提供了一项具有挑战性的评估。

英文摘要：

Large language models (LLMs) are increasingly used as role-playing agents, yet their capacity to faithfully and consistently portray version-specific characters -- for example, superheroes across comic and cinematic universes -- remains underexplored. Superhero canons such as Marvel and DC provide a rich testbed: decades of storytelling yield multiple incarnations of the same character with distinct histories, values, and moral codes. To study this problem, we introduce Beyond One World, a benchmark for character-grounded roleplay spanning 30 iconic heroes and 90 canon-specific versions. The benchmark comprises two tasks: (i) Canon Events, which probes factual recall of pivotal life stages, and (ii) Moral Dilemmas, which confronts models with ethically charged scenarios. We score responses for canonical accuracy and reasoning fidelity under a framework that separates internal deliberation ("thinking") from outward decisions ("acting"). We further propose Think-Act Matching, a metric that quantifies alignment between reasons and actions and serves as a proxy for model trustworthiness. Experiments across reasoning- and non-reasoning-oriented models yield three findings: (1) chain-of-thought prompting improves narrative coherence in weaker models but can reduce canonical accuracy in stronger ones; (2) cross-version generalization within a character remains a major obstacle; and (3) models often excel at either thinking or acting, but rarely both. Beyond One World exposes critical gaps in multiversal consistency and reasoning alignment, offering a challenging evaluation for role-playing LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。