arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.20828cs.AIcs.LG
收藏

如何衡量大型语言模型的智能?

How to Measure the Intelligence of Large Language Models?

Centre for Artificial Intelligence in Public Health Research(公共卫生研究人工智能中心);Robert Koch Institute(罗伯特·科赫研究所)

Nils Körber, Silvan Wehrli, Christopher Irrgang

AI总结:

本文探讨了大型语言模型智能的评估问题,指出仅靠任务特定统计指标不足以衡量,主张应从定性和定量两方面分别评估 LLMs 的智能。

摘要:

随着 ChatGPT 和其他大型语言模型(LLMs)的发布,关于当前和未来模型的智能、可能性及风险的讨论受到了广泛关注。这些讨论包括备受争议的所谓“超人类”AI 即将崛起的情景,即比人类聪明几个数量级的 AI 系统。秉承 Alan Turing 的精神,毫无疑问,当前最先进的语言模型已经通过了他著名的测试。此外,当前模型在几项基准测试中优于人类,因此公开可用的 LLMs 已经成为连接日常生活、工业和科学的多功能伙伴。尽管 LLMs 能力惊人,但有时在人类看来微不足道的任务上却完全失败。在其他情况下,LLMs 的可信度变得更加难以捉摸和评估。以学术界为例,语言模型能够仅凭少量输入就针对给定主题撰写出令人信服的研究文章。然而,AI 生成的文本在事实一致性方面缺乏可信度或存在持续幻觉,导致许多科学期刊对基于 AI 的内容实施了一系列限制。鉴于这些观察结果,人们开始讨论适用于人类智能的指标是否也能应用于计算方法。事实上,指标的选择已被证明会极大地影响对潜在智能涌现的评估。在此,我们认为 LLMs 的智能不仅应通过特定任务的统计指标来评估,还应从定性和定量指标两方面分别进行评估。

英文摘要:

With the release of ChatGPT and other large language models (LLMs) the discussion about the intelligence, possibilities, and risks, of current and future models have seen large attention. This discussion included much debated scenarios about the imminent rise of so-called "super-human" AI, i.e., AI systems that are orders of magnitude smarter than humans. In the spirit of Alan Turing, there is no doubt that current state-of-the-art language models already pass his famous test. Moreover, current models outperform humans in several benchmark tests, so that publicly available LLMs have already become versatile companions that connect everyday life, industry and science. Despite their impressive capabilities, LLMs sometimes fail completely at tasks that are thought to be trivial for humans. In other cases, the trustworthiness of LLMs becomes much more elusive and difficult to evaluate. Taking the example of academia, language models are capable of writing convincing research articles on a given topic with only little input. Yet, the lack of trustworthiness in terms of factual consistency or the existence of persistent hallucinations in AI-generated text bodies has led to a range of restrictions for AI-based content in many scientific journals. In view of these observations, the question arises as to whether the same metrics that apply to human intelligence can also be applied to computational methods and has been discussed extensively. In fact, the choice of metrics has already been shown to dramatically influence assessments on potential intelligence emergence. Here, we argue that the intelligence of LLMs should not only be assessed by task-specific statistical metrics, but separately in terms of qualitative and quantitative measures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑