SimulBench：通过创造性模拟任务评估语言模型
SimulBench: Evaluating Language Models with Creative Simulation Tasks
AI总结：
该研究推出评估大语言模型的创造性模拟任务基准SimulBench，采用固定LLM作为用户智能体生成多轮对话脚本、用GPT-4自动评估的框架，实验显示这类任务仍具挑战性，专有模型领先开源模型18.55%。
摘要：
我们推出了SimulBench，这是一个旨在通过多样化的创造性模拟场景（例如扮演Linux终端或与用户进行文字游戏）评估大语言模型（LLM）的基准。尽管这些模拟任务能够有效衡量LLM的通用智能，但它们很少被纳入现有基准中。该领域的一大挑战是开发一套评估框架，既能公平测试不同的LLM，又能保留用户与AI之间模拟任务的多轮交互特性。为解决这一问题，我们提出使用一个固定的LLM作为用户智能体，先与被测LLM交互，收集不同任务下的对话内容，再从中提取有难度的对话脚本，用于评估不同的目标LLM。为了实现对SimulBench的自动评估，我们采用GPT-4作为评估器，负责根据多轮对话脚本，审查目标LLM生成的最终回复的质量。我们的综合实验表明，这些模拟任务凭借其独特属性仍然构成重大挑战，同时也体现出专有模型与最先进的开源LLM之间的差距。例如，GPT-4-turbo的表现优于LLaMA-3-70b-Chat，胜出的案例占比高出18.55%。
英文摘要：
We introduce SimulBench, a benchmark designed to evaluate large language models (LLMs) across a diverse collection of creative simulation scenarios, such as acting as a Linux terminal or playing text games with users. While these simulation tasks serve as effective measures of an LLM's general intelligence, they are seldom incorporated into existing benchmarks. A major challenge is to develop an evaluation framework for testing different LLMs fairly while preserving the multi-round interactive nature of simulation tasks between users and AI. To tackle this issue, we suggest using a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks. Then, challenging dialogue scripts are extracted for evaluating different target LLMs. To facilitate automatic assessment on \DataName{}, GPT-4 is employed as the evaluator, tasked with reviewing the quality of the final response generated by the target LLMs given multi-turn dialogue scripts. Our comprehensive experiments indicate that these simulation tasks continue to pose a significant challenge with their unique natures and show the gap between proprietary models and the most advanced open LLMs. For example, GPT-4-turbo outperforms LLaMA-3-70b-Chat on 18.55\% more cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
