arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01360cs.CL
收藏

是什么让医学声明(不)可验证?面向事实验证的实体与关系属性分析

What Makes Medical Claims (Un)Verifiable? Analyzing Entity and Relation Properties for Fact Verification

University of Stuttgart(斯图加特大学);University of Bamberg(班贝格大学)

Amelie Wührl, Yarik Menchaca Resendiz, Lara Grimminger, Roman Klinger

AI总结:

本研究分析生物医学声明中实体与关系属性对可验证性的影响,构建了标注三元组、证据及核查结论的BEAR-Fact语料库,揭示否定声明难验证等发现,证明可通过声明文本估计证据检索成功率。

摘要:

生物医学声明验证若无法找到证据便会失败。这类情况下,事实核查结论仍未知,声明属于不可验证声明。为改进这一问题,我们需明确是否存在影响声明可验证性的属性。本研究假设实体和关系是生物医学声明结构的核心变量,分析二者的属性能否帮助区分可验证与不可验证声明。在一项由训练有素的标注专家参与的研究中,我们要求专家为生物医学声明寻找证据,并观察他们如何优化证据检索的搜索查询。由此构建了首个标注有主-谓-宾三元组、证据文档及事实核查结论的科学事实验证语料库(BEAR-Fact 语料库)。我们发现:(1)为否定式声明(如X不会导致Y)寻找证据尤其具有挑战性。此外,标注者主要通过增加搜索约束、将实体标准化为规范名称来处理查询。(2)我们将内部标注结果与由医学专家和普通大众参与的小型众包场景标注结果对比,发现领域专业知识对标注可靠性无显著影响。最后(3),我们证明仅通过声明文本就能可靠估计证据检索的成功率(F值为0.82),但识别不可验证声明的难度更高(F值为0.27)。该数据集可在http://www.ims.uni-stuttgart.de/data/bioclaim获取。

英文摘要:

Biomedical claim verification fails if no evidence can be discovered. In these cases, the fact-checking verdict remains unknown and the claim is unverifiable. To improve upon this, we have to understand if there are any claim properties that impact its verifiability. In this work we assume that entities and relations define the core variables in a biomedical claim's anatomy and analyze if their properties help us to differentiate verifiable from unverifiable claims. In a study with trained annotation experts we prompt them to find evidence for biomedical claims, and observe how they refine search queries for their evidence search. This leads to the first corpus for scientific fact verification annotated with subject-relation-object triplets, evidence documents, and fact-checking verdicts (the BEAR-Fact corpus). We find (1) that discovering evidence for negated claims (e.g., X-does-not-cause-Y) is particularly challenging. Further, we see that annotators process queries mostly by adding constraints to the search and by normalizing entities to canonical names. (2) We compare our in-house annotations with a small crowdsourcing setting where we employ medical experts and laypeople. We find that domain expertise does not have a substantial effect on the reliability of annotations. Finally, (3), we demonstrate that it is possible to reliably estimate the success of evidence retrieval purely from the claim text~(.82\F), whereas identifying unverifiable claims proves more challenging (.27\F). The dataset is available at http://www.ims.uni-stuttgart.de/data/bioclaim.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑