CRAB：生物医学中检索增强大语言模型策展能力评估基准
CRAB: A Benchmark for Evaluating Curation of Retrieval-Augmented LLMs in Biomedicine
AI总结：
针对生物医学检索增强大语言模型策展能力评估空白，提出多语言基准CRAB，采用基于引用的新指标，发现主流模型策展性能差异显著。
摘要：
检索增强大语言模型（LLMs）的最新发展在生物医学应用中展现出巨大潜力。然而，在可靠评估其策展能力方面仍存在关键空白，策展能力即模型选择并整合相关参考文献同时过滤噪声的过程。为解决这一问题，我们推出了生物医学检索增强大语言模型策展基准（CRAB），这是首个专为评估生物医学中检索增强大语言模型策展能力而设计的多语言基准，提供英语、法语、德语和中文版本。通过引入一种新颖的基于引用的评估指标，CRAB量化了生物医学中检索增强大语言模型的策展性能。实验结果显示，主流大语言模型的策展性能存在显著差异，凸显了在生物医学领域提升该能力的紧迫性。我们的数据集可在 https://huggingface.co/datasets/zhm0/CRAB 获取。
英文摘要：
Recent development in Retrieval-Augmented Large Language Models (LLMs) have shown great promise in biomedical applications. How ever, a critical gap persists in reliably evaluating their curation ability the process by which models select and integrate relevant references while filtering out noise. To address this, we introduce the benchmark for Curation of Retrieval-Augmented LLMs in Biomedicine (CRAB), the first multilingual benchmark tailored for evaluating the biomedical curation of retrieval-augmented LLMs, available in English, French, German and Chinese. By incorporating a novel citation-based evaluation metric, CRAB quantifies the curation performance of retrieval-augmented LLMs in biomedicine. Experimental results reveal significant discrepancies in the curation performance of mainstream LLMs, underscoring the urgent need to improve it in the domain of biomedicine. Our dataset is available at https://huggingface.co/datasets/zhm0/CRAB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
