arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.14879cs.CLcs.AI
收藏

ByteSized32:用于生成以文本游戏表达的任务特定世界模型的语料库与挑战任务

ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games

University of Arizona(亚利桑那大学);Microsoft Research Montréal(微软蒙特利尔研究院);New York University(纽约大学);Johns Hopkins University(约翰斯·霍普金斯大学)

Ruoyao Wang, Graham Todd, Eric Yuan, Ziang Xiao, Marc-Alexandre Côté, Peter Jansen

AI总结:

本研究调查语言模型生成科学与常识推理任务的显式交互式世界模型的能力,提出ByteSized32语料库,并实证GPT-4通过单次学习和自我反思提升生成可运行文本游戏的效果,引入自动化评估指标,作为世界建模与代码生成交汇领域的挑战任务。

摘要:

在本研究中,我们调查了语言模型生成针对科学与常识推理任务的显式、可解释且交互式世界模型的能力。我们将其操作化为一个生成文本游戏的任务,表达为数百行Python代码。为促进此任务,我们引入了ByteSized32(代码:github.com/cognitiveailab/BYTESIZED32),这是一个包含32个以推理为重点的文本游戏的语料库,总计2万行Python代码。我们实证表明,GPT-4能将这些游戏用作单次上下文学习的模板,在28%的情况下成功生成关于未见主题的可运行游戏。当允许其自我反思程序错误时,游戏可运行性大幅提升至57%。虽然评估模拟保真度耗时费力,但我们引入了一套自动化指标来评估游戏保真度、技术有效性、对任务规范的遵循度以及可获胜性,显示出与人类专家评分的高度一致性。我们将其作为一个挑战任务提出,以推动世界建模与代码生成交汇领域的进一步发展。

英文摘要:

In this work, we investigate the capacity of language models to generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. We operationalize this as a task of generating text games, expressed as hundreds of lines of Python code. To facilitate this task, we introduce ByteSized32 (Code: github.com/cognitiveailab/BYTESIZED32), a corpus of 32 reasoning-focused text games totaling 20k lines of Python code. We empirically demonstrate that GPT-4 can use these games as templates for single-shot in-context learning, successfully producing runnable games on unseen topics in 28% of cases. When allowed to self-reflect on program errors, game runnability substantially increases to 57%. While evaluating simulation fidelity is labor-intensive, we introduce a suite of automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability, showing a high degree of agreement with expert human ratings. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑