The rapid advancement of large language models (LLMs) has inspired researchers to integrate them extensively into the academic workflow, potentially reshaping how research is practiced and reviewed. While previous studies highlight the potential of LLMs in supporting research and peer review, their dual roles in the academic workflow and the complex interplay between research and review bring new risks that remain largely underexplored. In this study, we focus on how the deep integration of LLMs into both peer-review and research processes may influence scholarly fairness, examining the potential risks of using LLMs as reviewers by simulation. This simulation incorporates a research agent, which generates papers and revises, alongside a review agent, which assesses the submissions. Based on the simulation results, we conduct human annotations and identify pronounced misalignment between LLM-based reviews and human judgments: (1) LLM reviewers systematically inflate scores for LLM-authored papers, assigning them markedly higher scores than human-authored ones; (2) LLM reviewers persistently underrate human-authored papers with critical statements (e.g., risk, fairness), even after multiple revisions. Our analysis reveals that these stem from two primary biases in LLM reviewers: a linguistic feature bias favoring LLM-generated writing styles, and an aversion toward critical statements. These results highlight the risks and equity concerns posed to human authors and academic research if LLMs are deployed in the peer review cycle without adequate caution. On the other hand, revisions guided by LLM reviews yield quality gains in both LLM-based and human evaluations, illustrating the potential of the LLMs-as-reviewers for early-stage researchers and enhancing low-quality papers.
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
L3S Research Center, Leibniz University Hannover(莱布尼茨汉诺威大学L3S研究中心);Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School(布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所);Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)(下萨克森州医学人工智能与因果方法中心(CAIMed))
Large Language Models as 'Hidden Persuaders': Fake Product Reviews are Indistinguishable to Humans and Machines
N/LAB, Nottingham University Business School, University of Nottingham, UK(N/LAB,诺丁汉大学商学院,诺丁汉大学,英国);Haydn Green Institute for Entrepreneurship and Innovation, University of Nottingham, UK(创业与创新研究院,诺丁汉大学,英国);Strategic Innovation Ltd, UK(战略创新有限公司,英国)
A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews
Data Science Research Lab, Department of Statistics, University of Rajshahi(数据科学研究实验室,统计系,拉贾沙希大学);Department of Statistics, Mawlana Bhashani Science and Technology University(统计系,穆拉纳·巴希安尼科学与技术大学);Department of Statistics, University of Rajshahi(统计系,拉贾沙希大学)