Critical peer review of scientific manuscripts presents a significant challenge for Large Language Models (LLMs), partly due to data limitations and the complexity of expert reasoning. This report introduces Persistent Workflow Prompting (PWP), a potentially broadly applicable prompt engineering methodology designed to bridge this gap using standard LLM chat interfaces (zero-code, no APIs). We present a proof-of-concept PWP prompt for the critical analysis of experimental chemistry manuscripts, featuring a hierarchical, modular architecture (structured via Markdown) that defines detailed analysis workflows. We develop this PWP prompt through iterative application of meta-prompting techniques and meta-reasoning aimed at systematically codifying expert review workflows, including tacit knowledge. Submitted once at the start of a session, this PWP prompt equips the LLM with persistent workflows triggered by subsequent queries, guiding modern reasoning LLMs through systematic, multimodal evaluations. Demonstrations show the PWP-guided LLM identifying major methodological flaws in a test case while mitigating LLM input bias and performing complex tasks, including distinguishing claims from evidence, integrating text/photo/figure analysis to infer parameters, executing quantitative feasibility checks, comparing estimates against claims, and assessing a priori plausibility. To ensure transparency and facilitate replication, we provide full prompts, detailed demonstration analyses, and logs of interactive chats as supplementary resources. Beyond the specific application, this work offers insights into the meta-development process itself, highlighting the potential of PWP, informed by detailed workflow formalization, to enable sophisticated analysis using readily available LLMs for complex scientific tasks.
Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review
Shanghai Jiao Tong University(上海交通大学);Georgia Institute of Technology(佐治亚理工学院);Shanghai AI Laboratory(上海人工智能实验室);University of Georgia(佐治亚大学);Oxford University(牛津大学)
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learning
Chinese Academy of Sciences & City University of Hong Kong(中国科学院与香港城市大学);City University of Hong Kong(香港城市大学);Shenzhen University & Beijing Dongbi Data Technology(深圳大学与北京东碧数据技术);University of Chinese Academy of Sciences(中国科学院大学)
Large language models for automated scholarly paper review: A survey
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,科学与技术研究院);School of Economics and Management, Guizhou Normal University(经济学与管理学院,贵州师范大学);School of Artificial Intelligence, Guangzhou Institute of Science and Technology(人工智能学院,科学与技术研究院)
Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models
AllSci Corp(AllSci公司);Sunwater Capital(Sunwater资本);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);Dept. of Computer Science, Northern Illinois University(北伊利诺伊大学计算机科学系);Institute of Foundation Models, MBZUAI(基础模型研究所,MBZUAI)
Peering Inside the Black Box: Uncovering LLM Errors in Optimization Modelling through Component-Level Evaluation
Computer Science Department, KFUPM, Dhahran 31261, Saudi Arabia;SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum \& Minerals, Dhahran, 31261, Saudi Arabia
SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering
State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD与CG国家重点实验室);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Software Technology, Zhejiang University(浙江大学软件技术学院)
'Quis custodiet ipsos custodes?' Who will watch the watchmen? On Detecting AI-generated peer-reviews
Indian Institute of Technology Patna(印度理工学院巴特那分校);National Center for Computational Sciences, Oak Ridge National Laboratory(橡树岭国家实验室国家计算科学中心);School of AI and Data Science, IIT Jodhpur(焦特布尔印度理工学院人工智能与数据科学学院)
Prompting Multi-Modal Tokens to Enhance End-to-End Autonomous Driving Imitation Learning with LLMs
University of Technology Sydney(悉尼科技大学);Australia Artificial Intelligence Institute(澳大利亚人工智能研究院);School of Computer Science(计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Automated scholarly paper review: Concepts, technologies, and challenges
Xiamen University(厦门大学);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism(文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室)
Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators
Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney(澳大利亚人工智能研究所,工程与信息科技学院,新南威尔士大学);Max Planck Institute for Solid State Research(马克斯·普朗克固体物理研究所);Science Policy and Strategy Department, Administrative Headquarters of the Max Planck Society(马克斯·普朗克学会科学政策与战略部门,行政总部)
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
School of Economics and Management, Nanjing University of Science and Technology(南京理工大学经济管理学院);School of Electronic Engineering and Computer Science, Queen Mary University of London(伦敦玛丽女王大学电子工程与计算机科学学院);School of Management, Anhui University(安徽大学管理学院)
LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions
Singapore University of Technology and Design(新加坡科技设计大学);Nanyang Technological University(南洋理工大学);National University of Singapore(国立新加坡大学);Lambda Labs(Lambda实验室)
MOPRD: A multidisciplinary open peer review dataset
School of Informatics, Xiamen University(厦门大学信息学院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism(文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室);College of Foreign Languages and Cultures, Xiamen University(厦门大学外文学院)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)