GenAI4IDN Benchmark 3.0——用于评估交互式数字叙事设计生成式AI工具的公共工具
The GenAI4IDN Benchmark 3.0 - a Public Tool to Assess Generative AI Tools for the Design of Interactive Digital Narratives
浏览论文内容
中文总结 AI 辅助
本文提出GENAI4IDN Benchmark 3.0，一个基于Web的AI辅助评估框架，用于盲评生成式AI在交互式数字叙事中的创作能力，并新增事实核查与偏见检测任务，以应对模型局限。
中文摘要 AI 辅助
本文介绍了GENAI4IDN Benchmark 3.0，这是一个用于评估生成式AI工具在创建交互式数字叙事（IDN）方面的评估框架的第三次迭代。该迭代超越了手动测试，通过一个可公开访问的Web应用程序（https://this URL）引入了AI辅助评估，使社区能够按需运行基准测试、添加新模型并提出新任务。修订后的评估框架采用“盲评”以避免模型偏差，并能处理以前需要人工评分员的复杂媒体，如音乐、视频和完整的IDN。一个重要的新增内容——回应了ICIDS 2025期间提出的关切——是增加了事实核查和偏见检测，配有专门的任务和评分标准，并由在所述情境中具有亲身经验的人工评分员进行验证。来自多种模型的结果报告了不断成熟的创作能力，同时观察到思维失控和过度热心的安全过滤器等局限性。事实核查可靠地捕捉到了微妙的历史不准确、时代错误和捏造的主张，而偏见评分员则持续暴露了结构性假设、陈词滥调和对边缘群体的抹除。
英文摘要
This paper presents GENAI4IDN Benchmark 3.0, the third iteration of an evaluation framework to assess Generative AI tools for creating Interactive Digital Narratives (IDNs). Moving beyond manual testing, this iteration introduces AI-assisted evaluation through a publicly accessible web application (https://genai4idn.com), enabling the community to run benchmarks on demand, add new models, and propose new tasks. The revised evaluation framework is "blinded" to avoid model-bias, and can handle complex media such as music, videos, and full IDNs that previously required human raters. A significant addition - responding to concerns raised during ICIDS 2025 - is the addition of fact-checking and bias detection with dedicated tasks and rubrics, validated by human raters with lived experience in the depicted contexts. Findings from a diverse range of models report on maturing creative capabilities while observing runaway thinking and overzealous safety filters as limitations. Fact-checking reliably caught subtle historical inaccuracies, anachronisms, and fabricated claims while the bias rater consistently exposed structural assumptions, tropes, and marginalized group erasures.
发表机构
- Södertörn University（南泰尔恩大学）
- Stockholm University（斯德哥尔摩大学）
机构由 AI 辅助整理，请以论文原文为准。
