arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.03332cs.AIphysics.chem-ph
收藏

AI驱动的学术同行评审:基于持久工作流提示、元提示和元推理

AI-Driven Scholarly Peer Review via Persistent Workflow Prompting, Meta-Prompting, and Meta-Reasoning

Lobachevsky State University of Nizhny Novgorod(下诺夫哥罗德国立大学)

Evgeny Markhasin

AI总结:

本文提出持久工作流提示(PWP)方法,通过分层模块化提示和元推理编码专家评审流程,使标准LLM能进行多模态、系统化的化学手稿批判性评审,识别方法缺陷并执行复杂分析任务。

摘要:

科学手稿的批判性同行评审对大型语言模型(LLMs)提出了重大挑战,部分原因在于数据限制和专家推理的复杂性。本报告介绍了持久工作流提示(PWP),这是一种潜在广泛适用的提示工程方法,旨在利用标准LLM聊天界面(零代码、无API)弥合这一差距。我们提出了一个用于实验化学手稿批判性分析的概念验证PWP提示,其特点是具有分层、模块化架构(通过Markdown结构化),定义了详细的分析工作流。我们通过迭代应用元提示技术和元推理来开发此PWP提示,旨在系统化地编码专家评审工作流,包括隐性知识。该PWP提示在会话开始时提交一次,为LLM配备由后续查询触发的持久工作流,引导现代推理LLM进行系统性的多模态评估。演示表明,PWP引导的LLM在测试案例中识别出重大方法学缺陷,同时减轻了LLM输入偏差,并执行了复杂任务,包括区分主张与证据、整合文本/照片/图形分析以推断参数、执行定量可行性检查、将估计值与主张进行比较,以及评估先验合理性。为确保透明度和促进复现,我们提供了完整提示、详细演示分析和交互式聊天日志作为补充资源。除了具体应用之外,这项工作还提供了对元开发过程本身的见解,强调了PWP在详细工作流形式化的指导下,利用现成的LLM实现复杂科学任务的高级分析的潜力。

英文摘要:

Critical peer review of scientific manuscripts presents a significant challenge for Large Language Models (LLMs), partly due to data limitations and the complexity of expert reasoning. This report introduces Persistent Workflow Prompting (PWP), a potentially broadly applicable prompt engineering methodology designed to bridge this gap using standard LLM chat interfaces (zero-code, no APIs). We present a proof-of-concept PWP prompt for the critical analysis of experimental chemistry manuscripts, featuring a hierarchical, modular architecture (structured via Markdown) that defines detailed analysis workflows. We develop this PWP prompt through iterative application of meta-prompting techniques and meta-reasoning aimed at systematically codifying expert review workflows, including tacit knowledge. Submitted once at the start of a session, this PWP prompt equips the LLM with persistent workflows triggered by subsequent queries, guiding modern reasoning LLMs through systematic, multimodal evaluations. Demonstrations show the PWP-guided LLM identifying major methodological flaws in a test case while mitigating LLM input bias and performing complex tasks, including distinguishing claims from evidence, integrating text/photo/figure analysis to infer parameters, executing quantitative feasibility checks, comparing estimates against claims, and assessing a priori plausibility. To ensure transparency and facilitate replication, we provide full prompts, detailed demonstration analyses, and logs of interactive chats as supplementary resources. Beyond the specific application, this work offers insights into the meta-development process itself, highlighting the potential of PWP, informed by detailed workflow formalization, to enable sophisticated analysis using readily available LLMs for complex scientific tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑