DAHL：通过生物医学领域的基准数据集对长文本进行特定领域的自动化幻觉评估
DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine
AI总结：
本文提出DAHL基准数据集与自动化评估系统，通过将响应解构为原子单元计算DAHL分数，深入评估生物医学领域长文本生成中的事实冲突幻觉，实验发现模型规模超过70至80亿参数后事实准确性提升有限。
摘要：
我们引入了DAHL，这是一个旨在评估长文本生成中的幻觉的基准数据集和自动化评估系统，特别针对生物医学领域。我们的基准数据集从生物医学研究论文中精心整理而成，包含跨越29个类别的8,573个问题。DAHL通过将响应解构为原子单元（每个单元代表一条信息）来评估大型语言模型（LLMs）中的事实冲突幻觉。这些响应的准确率被平均以得出DAHL分数，与依赖多选题任务的先前方法相比，这提供了对幻觉更深入的评估。我们对8个不同模型进行了实验，发现较大的模型往往幻觉较少；然而，当模型规模超过70至80亿参数时，进一步扩展并不能显著提高事实准确性。DAHL分数有望成为人工标注偏好标签的有效替代方案，并能够扩展到其他专业领域。我们公开发布了数据集和代码。
英文摘要：
We introduce DAHL, a benchmark dataset and automated evaluation system designed to assess hallucination in long-form text generation, specifically within the biomedical domain. Our benchmark dataset, meticulously curated from biomedical research papers, consists of 8,573 questions across 29 categories. DAHL evaluates fact-conflicting hallucinations in Large Language Models (LLMs) by deconstructing responses into atomic units, each representing a single piece of information. The accuracy of these responses is averaged to produce the DAHL Score, offering a more in-depth evaluation of hallucinations compared to previous methods that rely on multiple-choice tasks. We conduct experiments with 8 different models, finding that larger models tend to hallucinate less; however, beyond a model size of 7 to 8 billion parameters, further scaling does not significantly improve factual accuracy. The DAHL Score holds potential as an efficient alternative to human-annotated preference labels, being able to be expanded to other specialized domains. We release the dataset and code in public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
