arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.06223cs.HCcs.AI
收藏

谁在思考?使用XAI手册推动以人为本的LLM评估

Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbook

Arthur(亚瑟)

Teresa Datta, John P. Dickerson

摘要:

部署的人工智能(AI)通常会影响人类,并且没有一个放之四海而皆准的指标来评估这些工具。以人为本的AI系统评估结合了定量和定性分析以及人工输入。它已经在可解释人工智能(XAI)和人机交互(HCI)社区中进行了深入的探索。差距仍然存在,但基本的理解是,人类与人工智能互动,以及随之而来的解释,以及人类的需求--包括他们的认知偏见和怪癖--应该被放在首位和中心,这一点被社区所接受。在本文中,我们将相对成熟的XAI领域与围绕大语言模型(LLM)的快速发展的研究热潮进行了比较。LLM的公认评价指标不是以人为中心的。我们认为,在过去十年中,XAI社区所走过的许多相同的道路将在讨论LLM时被翻新。具体来说,我们认为,人类的倾向-再一次,与他们的认知偏见和怪癖-应该休息的前面和中心时,评估部署的LLM。我们概述了以人为本的XAI评估的三个重点领域:心智模型,用例效用和认知参与,我们强调了探索LLM的每个概念的重要性。我们的目标是启动以人为中心的LLM评估。

英文摘要:

Deployed artificial intelligence (AI) often impacts humans, and there is no one-size-fits-all metric to evaluate these tools. Human-centered evaluation of AI-based systems combines quantitative and qualitative analysis and human input. It has been explored to some depth in the explainable AI (XAI) and human-computer interaction (HCI) communities. Gaps remain, but the basic understanding that humans interact with AI and accompanying explanations, and that humans' needs -- complete with their cognitive biases and quirks -- should be held front and center, is accepted by the community. In this paper, we draw parallels between the relatively mature field of XAI and the rapidly evolving research boom around large language models (LLMs). Accepted evaluative metrics for LLMs are not human-centered. We argue that many of the same paths tread by the XAI community over the past decade will be retread when discussing LLMs. Specifically, we argue that humans' tendencies -- again, complete with their cognitive biases and quirks -- should rest front and center when evaluating deployed LLMs. We outline three developed focus areas of human-centered evaluation of XAI: mental models, use case utility, and cognitive engagement, and we highlight the importance of exploring each of these concepts for LLMs. Our goal is to jumpstart human-centered LLM evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑