元评估本地大语言模型：重新思考严肃游戏的性能指标
Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Games
AI总结：
针对严肃游戏中开放式回应评估的主观性挑战，本研究调查了五个本地小型LLM在评估能源社区决策模拟游戏\textit{En-join}玩家回应时的可靠性，通过二元分类指标比较其性能，为AI评估工具的可信度提供见解。
摘要：
严肃游戏中开放式回应的评估面临独特挑战，因为正确性往往具有主观性。大型语言模型（LLMs）在这类场景中作为评估器的应用日益受到关注，但它们的准确性和一致性仍不确定，尤其是针对用于本地执行的小型模型。本研究调查了五个小规模LLM在评估\textit{En-join}（一款模拟能源社区决策的游戏）中玩家回应时的可靠性。通过利用传统的二元分类指标（包括准确性、真阳性率和真阴性率），我们在不同评估场景下系统地比较了这些模型。我们的结果凸显了每个模型的优势和局限性，揭示了敏感性、特异性和整体性能之间的权衡。我们证明，虽然一些模型在识别正确回应方面表现出色，但另一些模型在假阳性或不一致评估方面存在困难。研究结果强调，在部署LLM作为评估器时，需要情境感知的评估框架和谨慎的模型选择。这项工作为AI驱动评估工具的可信度这一更广泛的讨论做出了贡献，提供了关于不同LLM架构如何处理主观评估任务的见解。
英文摘要：
The evaluation of open-ended responses in serious games presents a unique challenge, as correctness is often subjective. Large Language Models (LLMs) are increasingly being explored as evaluators in such contexts, yet their accuracy and consistency remain uncertain, particularly for smaller models intended for local execution. This study investigates the reliability of five small-scale LLMs when assessing player responses in \textit{En-join}, a game that simulates decision-making within energy communities. By leveraging traditional binary classification metrics (including accuracy, true positive rate, and true negative rate), we systematically compare these models across different evaluation scenarios. Our results highlight the strengths and limitations of each model, revealing trade-offs between sensitivity, specificity, and overall performance. We demonstrate that while some models excel at identifying correct responses, others struggle with false positives or inconsistent evaluations. The findings highlight the need for context-aware evaluation frameworks and careful model selection when deploying LLMs as evaluators. This work contributes to the broader discourse on the trustworthiness of AI-driven assessment tools, offering insights into how different LLM architectures handle subjective evaluation tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
