大型语言模型、科学知识与事实性：一个简化人类专家评估的框架
Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation
AI总结：
本文提出一个分阶段评估框架，通过将流畅性、语义、事实性等任务分配给非专家与专家，系统评估了11个LLM在抗生素发现领域的科学事实编码能力，发现其事实准确性仍不足但存在改进趋势。
摘要：
本文介绍了一个用于评估事实性科学知识编码的框架，旨在简化通常由领域专家进行的人工评估过程。从在大规模科学文献语料库上训练的大型语言模型（LLMs）中进行推理并提取信息，有可能在生物医学发现领域带来阶跃式变革，降低获取和整合现有医学证据的障碍。本研究以抗生素发现为背景，探索了LLMs与生物医学背景知识进行对话的潜力。该框架包含三个评估步骤，每个步骤依次评估生成回答的不同方面：流畅性、提示对齐、语义连贯性、事实知识和特异性。通过将这些任务在非专家和专家之间进行划分，该框架减少了后者所需的工作量。本研究对十一个最先进的LLM模型（包括ChatGPT、GPT-4和Llama 2）在两个基于提示的任务中的能力进行了系统性评估：化合物定义生成和化合物-真菌关系判定。尽管近期模型在流畅性方面有所提升，但事实准确性仍然较低，且模型偏向于过度表示的实体。LLMs作为生物医学知识库的能力受到质疑，并强调了需要额外的系统性评估框架。虽然LLMs目前在零样本设置下尚不适合用作生物医学事实知识库，但随着模型变得领域专用化、规模扩大以及人类反馈水平提高，在事实性方向上出现了一种有前景的新兴特性。
英文摘要：
The paper introduces a framework for the evaluation of the encoding of factual scientific knowledge, designed to streamline the manual evaluation process typically conducted by domain experts. Inferring over and extracting information from Large Language Models (LLMs) trained on a large corpus of scientific literature can potentially define a step change in biomedical discovery, reducing the barriers for accessing and integrating existing medical evidence. This work explores the potential of LLMs for dialoguing with biomedical background knowledge, using the context of antibiotic discovery. The framework involves of three evaluation steps, each assessing different aspects sequentially: fluency, prompt alignment, semantic coherence, factual knowledge, and specificity of the generated responses. By splitting these tasks between non-experts and experts, the framework reduces the effort required from the latter. The work provides a systematic assessment on the ability of eleven state-of-the-art models LLMs, including ChatGPT, GPT-4 and Llama 2, in two prompting-based tasks: chemical compound definition generation and chemical compound-fungus relation determination. Although recent models have improved in fluency, factual accuracy is still low and models are biased towards over-represented entities. The ability of LLMs to serve as biomedical knowledge bases is questioned, and the need for additional systematic evaluation frameworks is highlighted. While LLMs are currently not fit for purpose to be used as biomedical factual knowledge bases in a zero-shot setting, there is a promising emerging property in the direction of factuality as the models become domain specialised, scale-up in size and level of human feedback.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
