“ChatGPT 比我的教授解释得更好吗?”:评估大型语言模型在对话中的解释能力并与人类基线进行比较
"Is ChatGPT a Better Explainer than My Professor?": Evaluating the Explanation Capabilities of LLMs in Conversation Compared to a Human Baseline
AI总结:
本研究利用5-Levels数据集和解释行为框架,评估GPT4在对话解释任务中的表现,比较人类解释者、GPT4标准回应及带解释动作的GPT4回应三种策略,以探究LLMs增强专家解释能力的潜力。
摘要:
解释构成知识共享的基础,并建立在沟通原则、社会动态和学习理论之上。我们特别关注对话式的解释方法,因为这种情境具有高度适应性和互动性。我们的研究利用了先前关于解释行为(explanatory acts)的工作,这是一个用于理解解释者和被解释者在对话中为解释、理解以及与对方互动而采用的不同策略的框架。我们使用了由Wachsmuth等人从WIRED YouTube系列构建的5-Levels数据集,该数据集后来由Booshehri等人用解释行为进行了标注。这些标注提供了一个框架,用于理解解释者和被解释者在构建回应时如何组织其反应。\n随着过去一年生成式人工智能的兴起,我们希望更好地理解大型语言模型(LLMs)的能力,以及它们如何在对话环境中增强专家解释者的能力。为实现这一目标,5-Levels数据集(我们使用Booshehri等人2023年带有解释行为标注的数据集)使我们能够审查LLMs参与解释对话的能力。为了评估LLMs生成解释者回应的有效性,我们比较了3种不同的策略,我们请人类标注者评估了3种不同的策略:人类解释者回应、GPT4标准回应、带有解释动作(Explanation Moves)的GPT4回应。
英文摘要:
Explanations form the foundation of knowledge sharing and build upon communication principles, social dynamics, and learning theories. We focus specifically on conversational approaches for explanations because the context is highly adaptive and interactive. Our research leverages previous work on explanatory acts, a framework for understanding the different strategies that explainers and explainees employ in a conversation to both explain, understand, and engage with the other party. We use the 5-Levels dataset was constructed from the WIRED YouTube series by Wachsmuth et al., and later annotated by Booshehri et al. with explanatory acts. These annotations provide a framework for understanding how explainers and explainees structure their response when crafting a response. With the rise of generative AI in the past year, we hope to better understand the capabilities of Large Language Models (LLMs) and how they can augment expert explainer's capabilities in conversational settings. To achieve this goal, the 5-Levels dataset (We use Booshehri et al.'s 2023 annotated dataset with explanatory acts.) allows us to audit the ability of LLMs in engaging in explanation dialogues. To evaluate the effectiveness of LLMs in generating explainer responses, we compared 3 different strategies, we asked human annotators to evaluate 3 different strategies: human explainer response, GPT4 standard response, GPT4 response with Explanation Moves.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。