社交推理游戏中大语言模型的细粒度与主题化评估
Fine-Grained and Thematic Evaluation of LLMs in Social Deduction Game
AI总结:
针对社交推理游戏中LLM评估的粗粒度和缺乏结构化错误分析问题,提出六个细粒度指标和主题分析,识别出四大推理失败类型。
摘要:
近期研究探讨了大语言模型(LLMs)是否能够支持模糊沟通,这种沟通的核心特征包括推断言外之意和规避怀疑。为开展该研究,研究者使用社交推理游戏(SDGs)作为实验环境,在此类游戏中玩家隐藏并推断特定信息。然而,先前工作往往忽视了在此类情境下应如何评估LLMs。具体而言,我们指出其采用的评估方法存在两个局限:第一,先前研究所使用的指标是粗粒度的,因为它们基于整体游戏结果,往往无法捕捉事件级行为;第二,错误分析缺乏结构化的方法论,难以产生能有效支撑评估结果的洞见。为解决这些局限,我们提出一种微观且系统化的研究途径。具体而言,我们引入了六个细粒度指标以解决第一个问题。针对第二个问题,我们进行了主题分析,识别出削弱LLMs在模糊沟通中表现的四大主要推理失败类型。
英文摘要:
Recent studies have investigated whether large language models (LLMs) can support obscured communication, which is characterized by core aspects such as inferring subtext and evading suspicions. To conduct the investigation, researchers have used social deduction games (SDGs) as their experimental environment, in which players conceal and infer specific information. However, prior work has often overlooked how LLMs should be evaluated in such settings. Specifically, we point out two limitations with the evaluation methods they employed. First, metrics used in prior studies are coarse-grained as they are based on overall game outcomes that often fail to capture event-level behaviors; Second, error analyses have lacked structured methodologies capable of producing insights that meaningfully support evaluation outcomes. To address these limitations, we propose a microscopic and systematic approach to the investigation. Specifically, we introduce six fine-grained metrics that resolve the first issue. To tackle the second issue, we conducted a thematic analysis and identified four major reasoning failures that undermine LLMs' performance in obscured communication.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。