DrugClaw与DrugAudit:基于原始来源的智能体与权威感知基准用于药物信息问答
DrugClaw and DrugAudit: A Primary-Source-Grounded Agent and Authority-Aware Benchmark for Drug-Information Question Answering
AI总结:
提出多智能体检索增强系统DrugClaw,通过反射驱动状态机查询药物注册与药物警戒知识库,并构建含3772条权威感知基准DrugAudit,在多个基准上取得最优性能。
摘要:
药物信息问答是一个高风险场景,其中虚构的事实可能误导临床决策,且每个引用事实的来源与事实本身同样重要。我们提出了DrugClaw,一个多智能体检索增强系统,通过反射驱动的状态机工作流查询药物注册和药物警戒技能库,并返回基于原始监管或同行评审记录的答案。我们还贡献了DrugAudit,一个包含3772个条目的权威感知基准,配备评估面板,在双评判者LLM作为评判者的协议下(评判者间kappa=0.88,几乎完美),对上游黄金来源匹配、令牌级语义片段重叠和引用忠实性进行评分。在DrugAudit以及MedQA(751)和PubMedQA(512)的药物相关子集上,DrugClaw在标题表的每一列均排名第一:两个评判者下的综合证据指数、评判者中介的答案正确性、原始来源率(0.918,比次优高10.1个百分点)、忠实性(0.887,高5.9个百分点)、MedQA(0.920)和PubMedQA(0.693)。
英文摘要:
Drug-information question answering is a high-stakes setting where hallucinated facts can mislead clinical decision-making and the provenance of each cited fact matters as much as the fact itself. We present DrugClaw, a multi-agent retrieval-augmented system that queries a registry of drug and pharmacovigilance skills via a reflection-driven state-machine workflow and returns answers grounded in primary regulatory or peer-reviewed records. We also contribute DrugAudit, a 3,772-item authority-aware benchmark with an evaluation panel that scores upstream-of-gold source match, token-level semantic snippet overlap, and citation faithfulness under a dual-judge LLM-as-judge protocol with inter-judge kappa = 0.88 (almost-perfect). Across DrugAudit plus drug-related subsets of MedQA (751) and PubMedQA (512), DrugClaw is top-1 on every column of the headline table: composite Evidence Index under both judges, judge-mediated answer correctness, primary-source rate (0.918, +10.1 pp over next-best), faithfulness (0.887, +5.9 pp), MedQA (0.920), and PubMedQA (0.693).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。