Orak：面向多样化视频游戏的LLM代理训练与评估基础基准
Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
AI总结：
Orak通过12种主流视频游戏构建基础基准，系统评估LLM代理在不同游戏场景中的能力，提供统一评估框架和细调数据集，推动多样化游戏代理发展。
摘要：
大型语言模型（LLM）代理正在重塑游戏行业，通过使更智能和更符合人类偏好的角色成为可能。然而，当前游戏基准测试无法满足实际需求：它们缺乏对各种游戏类型中LLM能力的评估，缺乏对复杂游戏玩法关键模块的研究，缺乏将预训练LLM适应为游戏代理的微调数据集。为填补这些空白，我们提出了Orak，一个涵盖12种流行视频游戏的基准测试，覆盖所有主要游戏类型。通过基于模型上下文协议（MCP）构建的即插即用接口，Orak支持在不同游戏场景中系统且可重复研究代理模块。我们进一步发布了涵盖多种游戏类型的专家LLM游戏轨迹微调数据集，将通用LLM转化为有效的游戏代理。Orak提供统一的评估框架，包括游戏排行榜、LLM战斗竞技场以及输入模态、代理策略和微调效果的消融研究，为多样化游戏代理奠定基础。代码和数据集可在https://github.com/krafton-ai/Orak和https://huggingface.co/datasets/KRAFTON/Orak获取。
英文摘要：
Large Language Model (LLM) agents are reshaping the game industry, by enabling more intelligent and human-preferable characters. Yet, current game benchmarks fall short of practical needs: they lack evaluations of diverse LLM capabilities across various game genres, studies of agentic modules crucial for complex gameplay, and fine-tuning datasets to adapt pre-trained LLMs into gaming agents. To fill these gaps, we present Orak, a benchmark for training and evaluating LLM agents across 12 popular video games spanning all major genres. Using a plug-and-play interface built on Model Context Protocol (MCP), Orak supports systematic and reproducible studies of agentic modules in varied game scenarios. We further release a fine-tuning dataset of expert LLM gameplay trajectories covering multiple genres, turning general LLMs into effective game agents. Orak offers a united evaluation framework, including game leaderboards, LLM battle arenas, and \fix{ablation studies} of input modality, agentic strategies, and fine-tuning effects, establishing a foundation towards versatile gaming agents. Code and datasets are available at https://github.com/krafton-ai/Orak and https://huggingface.co/datasets/KRAFTON/Orak.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
