Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. This article assesses which ChatGPT inputs (full text without tables, figures and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts. The results show that the optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones. Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing.
Authority Signals in AI Cited Health Sources: A Framework for Evaluating Source Credibility in ChatGPT Responses
York College, The City University of New York(纽约市立大学约克学院);Teachers College, Columbia University(哥伦比亚大学教师学院);William Paterson University(威廉帕特森大学);CUNY School of Public Health(纽约市立大学公共卫生学院)
Can AI be a Teaching Partner? Evaluating ChatGPT, Gemini, and DeepSeek across Three Teaching Strategies
Federal Institute of São Paulo (IFSP)(圣保罗联邦学院);Graduate School of Education of University of Pennsylvania (UPENN)(宾夕法尼亚大学教育研究生院);Institute of Mathematics and Computer Sciences of University of São Paulo (USP)(圣保罗大学数学与计算机科学研究所);Nucleus of Excellence in Social Technologies (NEES) of Federal University of Alagoas (UFAL)(阿拉戈斯联邦大学社会技术卓越中心)
Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation
School of Foreign Languages, Shanghai Jiao Tong University(上海交通大学外国语学院);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);Institute of Corpus Studies and Applications, Shanghai International Studies University(上海外国语大学语料库研究与应用研究所)
Luddy School of Informatics, Computing & Engineering(卢迪信息学、计算与工程学院);Center for Complex Networks & Systems(复杂网络与系统中心);Indiana University Network Science Institute(印第安纳大学网络科学研究所)
Can ChatGPT Really Understand Modern Chinese Poetry?
NLP;CT Lab, Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门科技大学CT实验室);Department of Portuguese, Faculty of Arts and Humanities, University of Macau(人文学院葡萄牙语系,澳门科技大学)
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
York University(约克大学);Nanyang Technological University(南洋理工大学);Dialpad Canada Inc.(Dialpad加拿大公司);Royal Bank of Canada(加拿大皇家银行);Salesforce Research(赛富时研究院)
ChatGPT as Data Augmentation for Compositional Generalization: A Case Study in Open Intent Detection
Ingenuity Labs Research Institute(创新实验室研究所);Queen’s University(女王大学);Smith School of Business(史密斯商学院);Department of Electrical and Computer Engineering(电气与计算机工程系)
ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?
Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington(印第安纳大学伯明顿分校勒迪信息、计算与工程学院);School of AI Convergence, Soongsil University(崇实大学人工智能融合学院);Soongsil University(崇实大学)
ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphs
University of Georgia(佐治亚大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);New York University Shanghai(上海纽约大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)