In this work, we investigate the capacity of language models to generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. We operationalize this as a task of generating text games, expressed as hundreds of lines of Python code. To facilitate this task, we introduce ByteSized32 (Code: github.com/cognitiveailab/BYTESIZED32), a corpus of 32 reasoning-focused text games totaling 20k lines of Python code. We empirically demonstrate that GPT-4 can use these games as templates for single-shot in-context learning, successfully producing runnable games on unseen topics in 28% of cases. When allowed to self-reflect on program errors, game runnability substantially increases to 57%. While evaluating simulation fidelity is labor-intensive, we introduce a suite of automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability, showing a high degree of agreement with expert human ratings. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.
ByteSized32Refactored: Towards an Extensible Interactive Text Games Corpus for LLM World Modeling and Evaluation
Johns Hopkins University(约翰霍普金斯大学);Liaoning Technical University(辽宁技术大学);Microsoft Research Montréal(微软研究院蒙特利尔分校);Central University of Finance and Economics(中央财经大学)
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
Olin College of Engineering(奥林工程学院);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Oxford(牛津大学);University College London(伦敦大学学院);ShanghaiTech University(上海科技大学);Columbia University(哥伦比亚大学);University of Manchester(曼彻斯特大学)
Video Games as a Corpus: Sentiment Analysis using Fallout New Vegas Dialog
University of Helsinki(赫尔辛基大学);École Normale Supérieure(巴黎高等师范学院)
我们提出了一种方法,用于从Fallout New Vegas中提取多语言情感注释对话数据集。游戏开发者已经在游戏中的每一行对话中预先注释了8种不同情感之一:\textit{anger, disgust, fear, happy, neutral, pained, sad }和\textit{surprised}。该游戏已被翻译成英语,西班牙语,德语,法语和意大利语。我们使用多语言BERT,XLMRoBERTa和语言特定的BERT模型提取的数据集上进行多语言,多标签情感分析的实验。在我们的实验中,对于大多数语言,多语言BERT的性能优于XLMRoBERTa,对于大多数语言,语言特定模型也略优于多语言BERT。最好的整体准确率为54%,它是通过使用西班牙语数据的多语言BERT。提取的数据集提出了一个具有挑战性的任务,情感分析。我们已经在Zenodo上公开发布了数据,包括测试和训练部分。由于版权原因,数据集被打乱了。
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Department of Psychiatry and Behavioral Sciences, University of Washington(华盛顿大学精神病学与行为科学系);Department of Psychiatry and Behavioral Sciences, Stanford University(斯坦福大学精神病学与行为科学系);Center for Biomedical Informatics Research, Stanford University(斯坦福大学生物医学信息学研究中心)