arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05367cs.CL
收藏

STRICTA:用于同行评审及更广泛领域的关键文本评估中的结构化推理

STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond

UKP Lab, ATHENE National Research Center for Applied Cybersecurity, Technical University of Darmstadt(UKP实验室,ATHENE国家应用网络安全研究中心,德累斯顿技术大学);AIML Lab, Hessian Center for AI, Technical University of Darmstadt(AIML实验室,黑森人工智能中心,德累斯顿技术大学);Synthetic RNA Biology, Technical University of Darmstadt(合成RNA生物学,德累斯顿技术大学)

Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, Iryna Gurevych

AI总结:

本文提出STRICTA框架,将关键文本评估建模为基于因果理论的逐步推理图,并构建了含4000多个推理步骤的专家数据集,实证研究专家推理并探讨LLM对专家工作流的模仿与支持,为人机协作推理奠定基础。

摘要:

关键文本评估是许多专家活动的核心,例如事实核查、同行评审和论文评分。然而,现有工作将关键文本评估视为黑盒问题,限制了可解释性和人机协作。为了弥合这一差距,我们引入了关键文本评估中的结构化推理(STRICTA),这是一种新颖的规范框架,将文本评估建模为显式的、逐步的推理过程。STRICTA 借鉴因果关系理论(Pearl, 1995),将评估分解为相互关联的推理步骤图。该图基于专家交互数据进行填充,并用于研究评估过程以及促进人机协作。我们正式定义了 STRICTA,并将其应用于生物医学论文评估的研究中,生成了一个包含来自约 40 位生物医学专家对 20 多篇论文的超过 4000 个推理步骤的数据集。我们使用该数据集实证研究关键文本评估中的专家推理,并调查 LLM 是否能够模仿和支持这些工作流程中的专家。由此产生的工具和数据集为研究同行评审及更广泛领域中文本评估的协作式专家-AI 推理铺平了道路。

英文摘要:

Critical text assessment is at the core of many expert activities, such as fact-checking, peer review, and essay grading. Yet, existing work treats critical text assessment as a black box problem, limiting interpretability and human-AI collaboration. To close this gap, we introduce Structured Reasoning In Critical Text Assessment (STRICTA), a novel specification framework to model text assessment as an explicit, step-wise reasoning process. STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory (Pearl, 1995). This graph is populated based on expert interaction data and used to study the assessment process and facilitate human-AI collaboration. We formally define STRICTA and apply it in a study on biomedical paper assessment, resulting in a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers. We use this dataset to empirically study expert reasoning in critical text assessment, and investigate if LLMs are able to imitate and support experts within these workflows. The resulting tools and datasets pave the way for studying collaborative expert-AI reasoning in text assessment, in peer review and beyond.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑