BioKGBench：面向生物医学科学的AI智能体知识图谱检查基准
BioKGBench: A Knowledge Graph Checking Benchmark of AI Agent for Biomedical Science
AI总结：
本文提出BioKGBench基准，将理解文献分解为科学声明验证和知识图谱问答两种原子能力，并设计KGCheck智能体任务，通过RAG识别知识图谱事实错误，发现现有智能体表现不足，同时提出BKGAgent基线并验证其有效性。
摘要：
追求面向生物医学科学的人工智能，即“AI科学家”，正吸引越来越多的关注，其中一种常见方法是构建由大语言模型（LLMs）驱动的副驾驶智能体。然而，为了评估此类系统，人们要么依赖对LLM本身的直接问答（QA），要么采用生物医学实验的方式。如何从AI科学家的视角精确地基准测试生物医学智能体，在很大程度上仍未得到探索。为此，我们从科学家最重要的能力之一——理解文献——中获得灵感，并引入了BioKGBench。与仅关注事实性问答的传统评估基准不同，在事实性问答中LLMs已知存在幻觉问题，我们首先将“理解文献”分解为两种原子能力：i）通过执行科学声明验证来“理解”研究论文中的非结构化文本，以及ii）与结构化知识图谱问答（KGQA）进行交互的能力，作为一种“文献” grounding 的形式。然后，我们制定了一个新颖的智能体任务，称为KGCheck，利用KGQA和基于领域的检索增强生成（RAG）来识别现有大规模知识图谱数据库中的事实性错误。我们为两个原子任务收集了两千多条数据，并为智能体任务收集了225条高质量标注数据。令人惊讶的是，我们发现最先进的智能体，无论是日常场景还是生物医学场景，在我们的基准测试上要么失败，要么表现较差。然后，我们引入了一个简单而有效的基线，称为BKGAgent。在广泛使用的流行知识图谱上，我们发现了超过90个事实性错误，这些错误为智能体进行发现提供了场景，并证明了我们方法的有效性。代码和数据可在 https://github.com/westlake-autolab/BioKGBench 获取。
英文摘要：
Pursuing artificial intelligence for biomedical science, a.k.a. AI Scientist, draws increasing attention, where one common approach is to build a copilot agent driven by Large Language Models (LLMs). However, to evaluate such systems, people either rely on direct Question-Answering (QA) to the LLM itself, or in a biomedical experimental manner. How to precisely benchmark biomedical agents from an AI Scientist perspective remains largely unexplored. To this end, we draw inspiration from one most important abilities of scientists, understanding the literature, and introduce BioKGBench. In contrast to traditional evaluation benchmark that only focuses on factual QA, where the LLMs are known to have hallucination issues, we first disentangle "Understanding Literature" into two atomic abilities, i) "Understanding" the unstructured text from research papers by performing scientific claim verification, and ii) Ability to interact with structured Knowledge-Graph Question-Answering (KGQA) as a form of "Literature" grounding. We then formulate a novel agent task, dubbed KGCheck, using KGQA and domain-based Retrieval-Augmented Generation (RAG) to identify the factual errors of existing large-scale knowledge graph databases. We collect over two thousand data for two atomic tasks and 225 high-quality annotated data for the agent task. Surprisingly, we discover that state-of-the-art agents, both daily scenarios and biomedical ones, have either failed or inferior performance on our benchmark. We then introduce a simple yet effective baseline, dubbed BKGAgent. On the widely used popular knowledge graph, we discover over 90 factual errors which provide scenarios for agents to make discoveries and demonstrate the effectiveness of our approach. The code and data are available at https://github.com/westlake-autolab/BioKGBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
