FDARxBench：基于FDA通用药物评估的监管与临床推理基准测试
FDARxBench: Benchmarking Regulatory and Clinical Reasoning on FDA Generic Drug Assessment
AI总结：
本文提出FDARxBench基准测试，用于评估基于文档的问答任务，通过FDA药物标签文档生成高质量问答示例，揭示语言模型在事实基础、长上下文检索和安全拒绝行为方面的差距。
摘要：
我们介绍了一个专家精心编纂、基于现实世界的基准测试，用于评估基于文档的问答（QA）任务，该任务受通用药物评估的启发，使用美国食品药品监督管理局（FDA）的药物标签文档。药物标签包含丰富但异质的临床和监管信息，使当前语言模型准确回答问题变得困难。通过与FDA监管评估员合作，我们引入了FDARxBench，并构建了一个多阶段流程，生成高质量、专家编纂的问答示例，涵盖事实性、多跳和拒绝任务，并设计评估协议以评估开放书和封闭书推理。在专有和开放权重模型上的实验揭示了事实基础、长上下文检索和安全拒绝行为之间的显著差距。虽然受FDA通用药物评估需求启发，该基准测试也提供了挑战监管级标签理解的坚实基础。该基准测试旨在支持对药物标签问题的LLM行为评估。
英文摘要：
We introduce an expert curated, real-world benchmark for evaluating document-grounded question-answering (QA) motivated by generic drug assessment, using the U.S. Food and Drug Administration (FDA) drug label documents. Drug labels contain rich but heterogeneous clinical and regulatory information, making accurate question answering difficult for current language models. In collaboration with FDA regulatory assessors, we introduce FDARxBench, and construct a multi-stage pipeline for generating high-quality, expert curated, QA examples spanning factual, multi-hop, and refusal tasks, and design evaluation protocols to assess both open-book and closed-book reasoning. Experiments across proprietary and open-weight models reveal substantial gaps in factual grounding, long-context retrieval, and safe refusal behavior. While motivated by FDA generic drug assessment needs, this benchmark also provides a substantial foundation for challenging regulatory-grade evaluation of label comprehension. The benchmark is designed to support evaluation of LLM behavior on drug-label questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
