arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10937cs.AIcs.CLcs.LG
收藏

理解“理解”:受大语言模型启发的实用框架

Understanding Understanding: A Pragmatic Framework Motivated by Large Language Models

AI21 Labs(AI21实验室);University of British Columbia(不列颠哥伦比亚大学);Stanford University(斯坦福大学)

Kevin Leyton-Brown, Yoav Shoham

AI总结:

针对大语言模型是否具备人类级理解能力的争议,提出了基于问答表现、通过随机抽样和概率置信边界判断智能体理解能力的实用框架,还发现附解释可降低样本复杂度,该框架可用于测试和构建有理解能力的AI。

摘要:

受大语言模型(Large Language Models,LLMs)的快速崛起以及关于其具备人类水平特质程度的争议启发,我们提出了一个用于测试任意智能体(无论是机器还是人类)是否理解某一主题的框架。该框架采用图灵测试的思路,仅基于智能体的表现,具体而言是其回答问题的水平来判断。框架的要素包括限定问题集合(即“理解范围”)、要求具备通用能力(“及格线”)、避免给出“荒谬答案”,但仍允许在部分问题上答错或给出“我不知道”的回答。要确定这些条件是否满足,需要对所有问题进行穷尽测试,这对于非平凡的理解范围而言是不可能实现的,但我们展示了如何通过随机抽样和应用概率置信边界来获得高置信度的判断结果。我们还证明,在答案之外附上解释能够降低达到可接受置信边界所需的样本复杂度,因为对答案的解释意味着智能体有能力回答许多相似的问题。根据我们的框架,当前的LLMs尚不能被认为理解非平凡领域,但由于该框架提供了一套测试理解能力的实用方案,它也可成为构建真正具备理解能力的AI智能体的工具。

英文摘要:

Motivated by the rapid ascent of Large Language Models (LLMs) and debates about the extent to which they possess human-level qualities, we propose a framework for testing whether any agent (be it a machine or a human) understands a subject matter. In Turing-test fashion, the framework is based solely on the agent's performance, and specifically on how well it answers questions. Elements of the framework include circumscribing the set of questions (the "scope of understanding"), requiring general competence ("passing grade"), avoiding "ridiculous answers", but still allowing wrong and "I don't know" answers to some questions. Reaching certainty about these conditions requires exhaustive testing of the questions which is impossible for nontrivial scopes, but we show how high confidence can be achieved via random sampling and the application of probabilistic confidence bounds. We also show that accompanying answers with explanations can improve the sample complexity required to achieve acceptable bounds, because an explanation of an answer implies the ability to answer many similar questions. According to our framework, current LLMs cannot be said to understand nontrivial domains, but as the framework provides a practical recipe for testing understanding, it thus also constitutes a tool for building AI agents that do understand.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑