受大语言模型(Large Language Models,LLMs)的快速崛起以及关于其具备人类水平特质程度的争议启发,我们提出了一个用于测试任意智能体(无论是机器还是人类)是否理解某一主题的框架。该框架采用图灵测试的思路,仅基于智能体的表现,具体而言是其回答问题的水平来判断。框架的要素包括限定问题集合(即“理解范围”)、要求具备通用能力(“及格线”)、避免给出“荒谬答案”,但仍允许在部分问题上答错或给出“我不知道”的回答。要确定这些条件是否满足,需要对所有问题进行穷尽测试,这对于非平凡的理解范围而言是不可能实现的,但我们展示了如何通过随机抽样和应用概率置信边界来获得高置信度的判断结果。我们还证明,在答案之外附上解释能够降低达到可接受置信边界所需的样本复杂度,因为对答案的解释意味着智能体有能力回答许多相似的问题。根据我们的框架,当前的LLMs尚不能被认为理解非平凡领域,但由于该框架提供了一套测试理解能力的实用方案,它也可成为构建真正具备理解能力的AI智能体的工具。
英文摘要:
Motivated by the rapid ascent of Large Language Models (LLMs) and debates about the extent to which they possess human-level qualities, we propose a framework for testing whether any agent (be it a machine or a human) understands a subject matter. In Turing-test fashion, the framework is based solely on the agent's performance, and specifically on how well it answers questions. Elements of the framework include circumscribing the set of questions (the "scope of understanding"), requiring general competence ("passing grade"), avoiding "ridiculous answers", but still allowing wrong and "I don't know" answers to some questions. Reaching certainty about these conditions requires exhaustive testing of the questions which is impossible for nontrivial scopes, but we show how high confidence can be achieved via random sampling and the application of probabilistic confidence bounds. We also show that accompanying answers with explanations can improve the sample complexity required to achieve acceptable bounds, because an explanation of an answer implies the ability to answer many similar questions. According to our framework, current LLMs cannot be said to understand nontrivial domains, but as the framework provides a practical recipe for testing understanding, it thus also constitutes a tool for building AI agents that do understand.
Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs
Nara Institute of Science and Technology(奈良科学技術大學);RIKEN(理化学研究所);Guardian Robot Project(守護機器人專案);Kyoto Institute of Technology(京都科技大學);Institute of Science Tokyo(東京科學研究所)
Max Planck Institute for Informatics Saarland Campus(马克斯·普朗克信息研究所萨尔兰州分校);Computer Science Saarland University(萨尔兰州大学计算机科学系);Max Planck Institute for Software Systems Saarland Campus(马克斯·普朗克软件系统研究所萨尔兰州分校)
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges
LMU Munich & Munich Center for Machine Learning(慕尼黑大学及慕尼黑机器学习中心);University of Cologne(科隆大学);University of Augsburg(奥格斯堡大学);Bosch Center for Artificial Intelligence(博世人工智能中心);Columbia University(哥伦比亚大学);University of Maryland, College Park(马里兰大学学院市分校)
Non-literal Understanding of Number Words by Language Models
Department of Linguistics, University of Tübingen(语言学系,图宾根大学);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学);University of Washington(华盛顿大学);Department of Psychology, Stanford University(心理学系,斯坦福大学);Departments of Psychology & Computer Science, Stanford University(心理学与计算机科学系,斯坦福大学)
Does GPT-4 surpass human performance in linguistic pragmatics?
Institute for Artificial Intelligence Research and Development of Serbia(塞尔维亚人工智能研究与发展研究所);University of Belgrade, Institute for Philosophy and Social Theory, Digital Society Lab(贝尔格莱德大学哲学与社会理论研究所数字社会实验室);Complexity Science Hub(复杂性科学中心);University of Novi Sad, Faculty of Philosophy(诺维萨德大学哲学学院);University of Kragujevac, Faculty of Engineering(克拉古耶瓦茨大学工程学院)
Presupposition and Reasoning in Conditionals: A Theory-Based Study of Humans and LLMs
Department of Cognitive Science, Carleton University(卡尔顿大学认知科学系);School of Computer Science, McGill University(麦吉尔大学计算机科学学院);Mila – Quebec AI Institute(魁北克人工智能研究所)
A fine-grained comparison of pragmatic language understanding in humans and language models
Massachusetts Institute of Technology(麻省理工学院);Sarah Lawrence College(莎拉·劳伦斯学院);Carleton University(卡尔顿大学);McGovern Institute for Brain Research, Massachusetts Institute of Technology(麻省理工学院麦戈文脑科学研究所)
Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMs
University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Mila, University of Montreal(蒙特利尔大学Mila研究所);Google DeepMind(谷歌DeepMind)
Pragmatically Informative Color Generation by Grounding Contextual Modifiers
Stanford University(斯坦福大学);Institute of High Performance Computing, Agency for Science, Technology and Research(科学技术研究局高性能计算研究所);National University of Singapore(新加坡国立大学)
City-VLM: Towards Multidomain Perception Scene Understanding via Multimodal Incomplete Learning
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Zhejiang University(浙江大学);Fudan University(复旦大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Terminus Technologies Co., Ltd.;Pengcheng Laboratory(鹏城实验室)