LitLLMs, LLMs for Literature Review: Are we there yet?
ServiceNow Research(ServiceNow研究院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);HEC Montreal(蒙特利尔高等商学院);University of Waterloo(滑铁卢大学);University of British Columbia(不列颠哥伦比亚大学);Polytechnique Montreal(蒙特利尔理工学院)
Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, Christopher Pal
文献综述是科学研究的核心组成部分,但撰写过程始终耗时且颇具挑战,近期科研论文数量的激增更是加剧了这一问题。本文探究了当前大语言模型(Large Language Models, LLMs)基于单篇摘要辅助撰写文献综述的零样本能力。我们将该任务拆解为两个模块:1. 给定查询摘要,检索相关研究工作;2. 基于检索结果撰写文献综述,并分析了LLMs在两个模块中的实际效能。\n在检索模块,我们提出了一种新颖的两步搜索策略:首先利用LLM从论文摘要中提取具有检索价值的关键词,再通过查询外部知识库获取潜在相关的论文。此外,我们研究了一种带归因机制的基于提示的重排序方法,实验表明,与朴素搜索方法相比,重排序将归一化召回率提升了一倍,同时还能为理解LLM的决策过程提供参考。\n在生成阶段,我们提出了两步生成方法:首先为综述拟定写作大纲,再按照大纲的步骤执行生成实际的文献综述内容。为评估不同基于LLM的文献综述方法,我们采用一套专为滚动适配新发布LLM设计的协议,从arXiv论文中构建了测试集,以避免零样本评估中的测试集污染问题。我们公开了该评估协议,以推动该领域的进一步研究与开发。实证结果表明,当将文献综述任务拆解为检索和规划等更小的子模块时,LLMs展现出了可观的应用潜力。包含演示系统和工具包的项目页面可通过以下链接访问:https://litllm.github.io。
英文摘要:
Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially due to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: 1. Retrieving related works given a query abstract, and 2. Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods, while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Our project page including a demonstration system and toolkit can be accessed here: https://litllm.github.io.
Large language models for automated scholarly paper review: A survey
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,科学与技术研究院);School of Economics and Management, Guizhou Normal University(经济学与管理学院,贵州师范大学);School of Artificial Intelligence, Guangzhou Institute of Science and Technology(人工智能学院,科学与技术研究院)
Stochastic Parrots or ICU Experts? Large Language Models in Critical Care Medicine: A Scoping Review
National Institute of Health Data Science, Peking University(北京大学健康医疗大数据国家研究院);Institute of Medical Technology, Peking University Health Science Center(北京大学医学部医学技术研究院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);Advanced Institute of Information Technology, Peking University(北京大学信息技术高等研究院);Peking University Third Hospital(北京大学第三医院);University of Liverpool(利物浦大学);Johns Hopkins School of Medicine(约翰斯·霍普金斯大学医学院);University of California, Los Angeles(加州大学洛杉矶分校);Peking University People’s Hospital(北京大学人民医院)
Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review
Peking University(北京大学);Peking University Cancer Hospital and Institute(北京大学肿瘤医院暨北京市肿瘤防治研究所);National Engineering Laboratory for Big Data Analysis and Applications, Peking University(北京大学大数据分析与应用技术国家工程实验室);Peking University Changsha Institute for Computing and Digital Economy(北京大学长沙计算与数字经济研究院)
Bridge-AI Lab(Bridge-AI实验室);University of Central Florida(中佛罗里达大学);Auburn University(奥本大学);Wichita State University(威奇托州立大学);Gustavus Adolphus College(古斯塔夫斯阿道夫学院)
Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review
Shanghai Jiao Tong University(上海交通大学);Georgia Institute of Technology(佐治亚理工学院);Shanghai AI Laboratory(上海人工智能实验室);University of Georgia(佐治亚大学);Oxford University(牛津大学)
Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges
Kyoto University(京都大学);Indiana University(印第安纳大学);National Taiwan Normal University(台湾师范大学);Georgia Institute of Technology(佐治亚理工学院);Hong Kong University of Science(香港科学大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Cornell University(康奈尔大学);University of Liverpool, UK(利物浦大学);University of Edinburgh, UK(爱丁堡大学);Zhejiang University(浙江大学);Purdue University(普渡大学);Emory University(埃默里大学);West China Biomedical Big Data Center, West China Hospital, Sichuan University(四川大学西昌生物大数据中心、西昌医院)
A Review of Large Language Models and Autonomous Agents in Chemistry
FutureHouse Inc.(FutureHouse公司);University of Rochester(罗切斯特大学);School of Chemistry and Materials Science(化学与材料科学学院);Rochester Institute of Technology(罗切斯特理工学院)
A Concise Review of Hallucinations in LLMs and their Mitigation
Computer Engineering K J Somaiya Institute of Technology Mumbai, India;Data Science K J Somaiya Institute of Technology Mumbai, India;Information Technology K J Somaiya Institute of Technology Mumbai, India
Building Trust in Conversational AI: A Comprehensive Review and Solution Architecture for Explainable, Privacy-Aware Systems using LLMs and Knowledge Graph
CeADAR - Ireland’s Centre for Applied AI, University College Dublin(爱尔兰应用人工智能中心CeADAR、都柏林大学学院)
Review-LLM: Harnessing Large Language Models for Personalized Review Generation
School of New Media Communication, Tianjin University(天津大学新媒体与传播学院);Du Xiaoman Financial(度小满金融);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部)
Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications
Renmin University of China(中国人民大学);The Chinese University of Hong Kong(香港中文大学);Shenzhen University(深圳大学);City University of Hong Kong(香港城市大学);Peking University(北京大学);Massachusetts General Hospital and Harvard Medical School(麻省总医院和哈佛医学院)
A Sentiment Consolidation Framework for Meta-Review Generation
School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review
University of Pittsburgh(匹兹堡大学);University of Pittsburgh Medical Center(匹兹堡大学医学中心);University of Texas Health Houston(德克萨斯大学休斯顿健康科学中心);Cleveland Clinic(克利夫兰诊所);Keck School of Medicine, University of Southern California(南加州大学凯克医学院);Weill Cornell Medicine(威尔康奈尔医学院)
CycleResearcher: Improving Automated Research via Automated Review
Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);School of Engineering, Westlake University(西湖大学工程学院);Zhejiang University(浙江大学);William&Mary(威廉玛丽学院);University College London(伦敦大学学院)
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Salesforce Research(Salesforce研究院);Hong Kong Polytechnic University(香港理工大学);Penn State University(宾夕法尼亚州立大学);Google DeepMind(谷歌DeepMind);University of Southern California(南加州大学);University of California, Davis(加州大学戴维斯分校);University of Basel(巴塞尔大学);Xiaomi AI Lab(小米AI实验室);Scale AI;HKUST(香港科技大学);Georgia Institute of Technology(佐治亚理工学院);University of California, Santa Cruz(加州大学圣克鲁兹分校);Singapore University of Technology and Design(新加坡科技设计大学);Texas A&M University(德克萨斯农工大学);Massey University(梅西大学);University of Liverpool(利物浦大学)
Do LLMs Understand Wine Descriptors Across Cultures? A Benchmark for Cultural Adaptations of Wine Reviews
Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系);Department of Food and Resource Economics, University of Copenhagen(哥本哈根大学食品与资源经济学系);Department of Food Science, University of Copenhagen(哥本哈根大学食品科学系);Department of Computer Science, Aarhus University(阿arhus大学计算机科学系)
Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models
AllSci Corp(AllSci公司);Sunwater Capital(Sunwater资本);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);Dept. of Computer Science, Northern Illinois University(北伊利诺伊大学计算机科学系);Institute of Foundation Models, MBZUAI(基础模型研究所,MBZUAI)