This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4, a state-of-the-art artificial intelligence language model, across multiple iterations, time spans and stylistic variations. The model rated responses to tasks within the Higher Education (HE) subject domain of macroeconomics in terms of their content and style. Statistical analysis was conducted in order to learn more about the interrater reliability, consistency of the ratings across iterations and the correlation between ratings in terms of content and style. The results revealed a high interrater reliability with ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting that GPT-4 is capable of generating consistent ratings across repetitions with a clear prompt. Style and content ratings show a high correlation of 0.87. When applying a non-adequate style the average content ratings remained constant, while style ratings decreased, which indicates that the large language model (LLM) effectively distinguishes between these two criteria during evaluation. The prompt used in this study is furthermore presented and explained. Further research is necessary to assess the robustness and reliability of AI models in various use cases.
Multi-LLM Thematic Analysis with Dual Reliability Metrics: Combining Cohen's Kappa and Semantic Similarity for Qualitative Research Validation
AZA Lab, Department of Psychiatry, Yale University(AZA实验室、精神病学系、耶鲁大学);Department of Psychiatry and Behavioral Sciences, Center for Collective Healing, Howard University(精神病学与行为科学系、集体愈合中心、霍华德大学);Department of Psychology, University of Exeter(心理学系、埃克塞特大学)
High-Throughput Phenotyping of Clinical Text Using Large Language Models
Dept. of Neurology \& Rehabilitation University of Illinois at Chicago Chicago, IL USA;Engineering Program Missouri State University Springfield, MO USA
Improving Data Efficiency via Curating LLM-Driven Rating Systems
University of California, Santa Cruz(加州大学圣克鲁兹分校);Center for Advanced AI, Accenture(埃森哲高级人工智能中心);BIAI, ZJUT(浙江工业大学BIAI);D5Data.ai;The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale
Ira A. Fulton Schools of Engineering Arizona State University(亚利桑那州立大学工程学院);Heinz College of Information Systems and Public Policy Carnegie Mellon University(卡内基梅隆大学信息系统与公共政策学院);School of Education and Counseling Psychology Santa Clara University(圣克拉拉大学教育与咨询心理学学院)
Emergency Department Revisit Quality Review Screening: Exploring Human Decision-Making and Artificial Intelligence Support
OSF HealthCare(OSF医疗保健);University of Illinois College of Medicine at Peoria(伊利诺伊大学皮奥里亚医学院);Saint Anthony College of Nursing(圣安东尼护理学院);Thomas Jefferson University's Sidney Kimmel College of Medicine(托马斯·杰斐逊大学西德尼·金梅尔医学院);Keylog Solutions LLC(Keylog解决方案有限责任公司)
A Multi-Instance LLM Framework for Rating the Quality of Surgical Feedback
Computing + Mathematical Sciences, California Institute of Technology(加州理工学院计算与数学科学系);Department of Urology, Cedars-Sinai(塞斯医疗中心泌尿科);Keck School of Medicine, University of Southern California(美国南加州大学凯克医学院)
Automated scoring of the Ambiguous Intentions Hostility Questionnaire using fine-tuned large language models
University of California, Los Angeles(加州大学洛杉矶分校);University of Texas at Tyler(德克萨斯大学泰勒分校);JFK Johnson Rehabilitation Institute(肯尼迪约翰逊康复研究所);University of Chicago(芝加哥大学)
Blinded Multi-Rater Comparative Evaluation of a Large Language Model and Clinician-Authored Responses in CGM-Informed Diabetes Counseling
Institute of Health Informatics, University College London(健康信息学研究所,伦敦大学学院);University College London Hospitals NHS Foundation Trust(伦敦大学学院医院 NHS 基础信托);Dartford and Gravesham NHS Foundation Trust(达特福德和格拉夫萨姆 NHS 基础信托);Royal Free London NHS Foundation Trust(伦敦皇家自由 NHS 基础信托)
Math anxiety and associative knowledge structure are entwined in psychology students but not in Large Language Models like GPT-3.5 and GPT-4o
Department of Psychology and Cognitive Science, University of Trento(心理学与认知科学系,特伦托大学);CogNosco Lab, Department of Psychology and Cognitive Science, University of Trento(CogNosco实验室,心理学与认知科学系,特伦托大学);Technical University of Munich(慕尼黑技术大学);Munich Data Science Institute(慕尼黑数据科学研究所);Centre for Change and Complexity in Learning, University of South Australia(学习变革与复杂性研究中心,澳大利亚南澳大利亚大学)