STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond
UKP Lab, ATHENE National Research Center for Applied Cybersecurity, Technical University of Darmstadt(UKP实验室,ATHENE国家应用网络安全研究中心,德累斯顿技术大学);AIML Lab, Hessian Center for AI, Technical University of Darmstadt(AIML实验室,黑森人工智能中心,德累斯顿技术大学);Synthetic RNA Biology, Technical University of Darmstadt(合成RNA生物学,德累斯顿技术大学)
Nils Dycke, Matej Zečević, Ilia Kuznetsov, Beatrix Suess, Kristian Kersting, Iryna Gurevych
Critical text assessment is at the core of many expert activities, such as fact-checking, peer review, and essay grading. Yet, existing work treats critical text assessment as a black box problem, limiting interpretability and human-AI collaboration. To close this gap, we introduce Structured Reasoning In Critical Text Assessment (STRICTA), a novel specification framework to model text assessment as an explicit, step-wise reasoning process. STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory (Pearl, 1995). This graph is populated based on expert interaction data and used to study the assessment process and facilitate human-AI collaboration. We formally define STRICTA and apply it in a study on biomedical paper assessment, resulting in a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers. We use this dataset to empirically study expert reasoning in critical text assessment, and investigate if LLMs are able to imitate and support experts within these workflows. The resulting tools and datasets pave the way for studying collaborative expert-AI reasoning in text assessment, in peer review and beyond.
Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt(普遍知识处理实验室(UKP实验室)、计算机科学系和海斯曼人工智能中心(hessian.AI)、达姆施塔特技术大学);Department of Computer Science at Queens College, City University of New York (CUNY)(纽约城市大学皇后学院计算机科学系)
AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
Department of Computer Science, Sapienza University of Rome(计算机科学系,罗马萨皮恩扎大学);Department of Computer, Control, and Management Engineering, Sapienza University of Rome(计算机、控制与管理工程系,罗马萨皮恩扎大学);eZecute S.R.L.(eZecute公司)
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
L3S Research Center, Leibniz University Hannover(莱布尼茨汉诺威大学L3S研究中心);Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School(布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所);Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)(下萨克森州医学人工智能与因果方法中心(CAIMed))
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback
UKP Lab, TU Darmstadt and Hessian Center for AI (hessian.AI)(图林人工智能实验室、德累斯顿理工大学和黑森人工智能中心);MBZUAI(马克斯·普朗克人工智能研究所);The Allen Institute for AI (AI2)(人工智能研究院)
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
School of Economics and Management, Nanjing University of Science and Technology(南京理工大学经济管理学院);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦玛丽女王大学电子工程与计算机科学学院);School of Management, Anhui University(安徽大学管理学院)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔高等商学院);ServiceNow Research(ServiceNow研究);Université de Montréal(蒙特利尔大学);Canada CIFAR Chair(加拿大 CIFAR 席位);Polytechnique Montréal(蒙特利尔理工学院)
本文提出 ReviewerToo 框架,用于研究 AI 辅助同行评审,在 ICLR 2025 数据集上验证其性能接近人类平均水平,并指出 AI 的优势与局限,提出集成指南,以提升评审的一致性和公平性。
PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Wenge Technology(文英科技)