Digital Socrates：通过解释性批判评估大语言模型
Digital Socrates: Evaluating LLMs through Explanation Critiques
AI总结：
针对大语言模型解释质量难以评估的问题，研究提出解释批判新任务，构建人工验证数据集，训练出开源自动批判模型Digital Socrates，可无需高价API和人工标注，自动精细评估模型解释能力，填补了相关评估工具空白。
摘要：
尽管大语言模型（LLM）能够在给出答案的同时提供推理解释，但人们对这些解释的本质和质量仍知之甚少。为此，我们的目标是定义一种表征现代模型解释能力的详细方法，并打造一款精细、可解释的解释评估工具，该工具无需依赖成本高昂的API调用或人工标注，就能自动生成此类表征。我们的研究路径包括：（a）定义“解释批判”这一新任务——识别并归类解释中的各类主要缺陷，同时提出针对性的改进建议；（b）为该任务构建一个规模可观、经人工验证的数据集；（c）利用该数据集训练一款名为Digital Socrates的开源自动批判模型。通过定量与定性分析，我们证明了Digital Socrates可通过检验学生模型的推理链，揭示关于这些模型的深度洞见，并且它首次实现了对模型解释的高质量、精细化自动评估。因此，Digital Socrates填补了用于理解和改进模型解释行为的评估工具领域的一项重要空白。
英文摘要：
While LLMs can provide reasoned explanations along with their answers, the nature and quality of those explanations are still poorly understood. In response, our goal is to define a detailed way of characterizing the explanation capabilities of modern models and to create a nuanced, interpretable explanation evaluation tool that can generate such characterizations automatically, without relying on expensive API calls or human annotations. Our approach is to (a) define the new task of explanation critiquing - identifying and categorizing any main flaw in an explanation and providing suggestions to address the flaw, (b) create a sizeable, human-verified dataset for this task, and (c) train an open-source, automatic critique model (called Digital Socrates) using this data. Through quantitative and qualitative analysis, we demonstrate how Digital Socrates is useful for revealing insights about student models by examining their reasoning chains, and how it can provide high-quality, nuanced, automatic evaluation of those model explanations for the first time. Digital Socrates thus fills an important gap in evaluation tools for understanding and improving the explanation behavior of models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
