Efficacy of Large Language Models in Systematic Reviews
Illinois Mathematics and Science Academy（伊利诺伊数学与科学学院）；Rick Reedy High School（里克·里迪高中）
Aaditya Shah, Shridhar Mehendale, Siddha Kanthi
备注：
Both Shah and Mehendale contributed equally to this work; order of authorship is random. This paper will be published in the proceedings of The 2nd International Conference on Foundation and Large Language Models (FLLM2024) in IEEE Xplore
This study investigates the effectiveness of Large Language Models (LLMs) in interpreting existing literature through a systematic review of the relationship between Environmental, Social, and Governance (ESG) factors and financial performance. The primary objective is to assess how LLMs can replicate a systematic review on a corpus of ESG-focused papers. We compiled and hand-coded a database of 88 relevant papers published from March 2020 to May 2024. Additionally, we used a set of 238 papers from a previous systematic review of ESG literature from January 2015 to February 2020. We evaluated two current state-of-the-art LLMs, Meta AI's Llama 3 8B and OpenAI's GPT-4o, on the accuracy of their interpretations relative to human-made classifications on both sets of papers. We then compared these results to a "Custom GPT" and a fine-tuned GPT-4o Mini model using the corpus of 238 papers as training data. The fine-tuned GPT-4o Mini model outperformed the base LLMs by 28.3% on average in overall accuracy on prompt 1. At the same time, the "Custom GPT" showed a 3.0% and 15.7% improvement on average in overall accuracy on prompts 2 and 3, respectively. Our findings reveal promising results for investors and agencies to leverage LLMs to summarize complex evidence related to ESG investing, thereby enabling quicker decision-making and a more efficient market.
Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages
Trinity Centre for Global Health（三一全球健康中心）；School of Psychology（心理学院）；Trinity College Dublin（都柏林圣三一学院）；University of York（约克大学）
Enhancing Systematic Reviews with Large Language Models: Using GPT-4 and Kimi
Pearson（培生集团）；Measurement Incoporated（测量公司）；British Columbia College of Nurses and Midwives（不列颠哥伦比亚省护士与助产士学院）；University of Illinois Springfield（伊利诺伊大学斯普林菲尔德分校）；HSHS Saint John’s Hospital（HSHS圣约翰医院）
本研究评估了 GPT-4 和 Kimi 两种大型语言模型在系统评价中的表现，发现其性能受数据量和问题复杂性的影响。
Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approach
Duke University（杜克大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Duke University School of Medicine（杜克大学医学院）；Duke Clinical Research Institute（杜克临床研究所）
Machine Learning Information Retrieval and Summarisation to Support Systematic Review on Outcomes Based Contracting
University of Warwick（华威大学）；Alan Turing Institute for Data Science and AI（艾伦·图灵数据科学与人工智能研究所）；Cranfield University（克兰菲尔德大学）；Government Outcomes Lab（政府成果实验室）；Blavatnik School of Government（布拉瓦尼克政府学院）；University of Oxford（牛津大学）
