arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.15249cs.CLcs.AIcs.DLcs.LG
收藏

LitLLMs,面向文献综述的大语言模型:我们做到了吗?

LitLLMs, LLMs for Literature Review: Are we there yet?

ServiceNow Research(ServiceNow研究院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);HEC Montreal(蒙特利尔高等商学院);University of Waterloo(滑铁卢大学);University of British Columbia(不列颠哥伦比亚大学);Polytechnique Montreal(蒙特利尔理工学院)

Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, Christopher Pal

AI总结:

该研究将文献综述任务拆解为检索与生成模块,提出两步检索重排策略和两步生成方法,构建适配新LLM的零样本评估测试集,验证了LLMs辅助撰写文献综述的潜力。

摘要:

文献综述是科学研究的核心组成部分,但撰写过程始终耗时且颇具挑战,近期科研论文数量的激增更是加剧了这一问题。本文探究了当前大语言模型(Large Language Models, LLMs)基于单篇摘要辅助撰写文献综述的零样本能力。我们将该任务拆解为两个模块:1. 给定查询摘要,检索相关研究工作;2. 基于检索结果撰写文献综述,并分析了LLMs在两个模块中的实际效能。\n在检索模块,我们提出了一种新颖的两步搜索策略:首先利用LLM从论文摘要中提取具有检索价值的关键词,再通过查询外部知识库获取潜在相关的论文。此外,我们研究了一种带归因机制的基于提示的重排序方法,实验表明,与朴素搜索方法相比,重排序将归一化召回率提升了一倍,同时还能为理解LLM的决策过程提供参考。\n在生成阶段,我们提出了两步生成方法:首先为综述拟定写作大纲,再按照大纲的步骤执行生成实际的文献综述内容。为评估不同基于LLM的文献综述方法,我们采用一套专为滚动适配新发布LLM设计的协议,从arXiv论文中构建了测试集,以避免零样本评估中的测试集污染问题。我们公开了该评估协议,以推动该领域的进一步研究与开发。实证结果表明,当将文献综述任务拆解为检索和规划等更小的子模块时,LLMs展现出了可观的应用潜力。包含演示系统和工具包的项目页面可通过以下链接访问:https://litllm.github.io。

英文摘要:

Literature reviews are an essential component of scientific research, but they remain time-intensive and challenging to write, especially due to the recent influx of research papers. This paper explores the zero-shot abilities of recent Large Language Models (LLMs) in assisting with the writing of literature reviews based on an abstract. We decompose the task into two components: 1. Retrieving related works given a query abstract, and 2. Writing a literature review based on the retrieved results. We analyze how effective LLMs are for both components. For retrieval, we introduce a novel two-step search strategy that first uses an LLM to extract meaningful keywords from the abstract of a paper and then retrieves potentially relevant papers by querying an external knowledge base. Additionally, we study a prompting-based re-ranking mechanism with attribution and show that re-ranking doubles the normalized recall compared to naive search methods, while providing insights into the LLM's decision-making process. In the generation phase, we propose a two-step approach that first outlines a plan for the review and then executes steps in the plan to generate the actual review. To evaluate different LLM-based literature review methods, we create test sets from arXiv papers using a protocol designed for rolling use with newly released LLMs to avoid test set contamination in zero-shot evaluations. We release this evaluation protocol to promote additional research and development in this regard. Our empirical results suggest that LLMs show promising potential for writing literature reviews when the task is decomposed into smaller components of retrieval and planning. Our project page including a demonstration system and toolkit can be accessed here: https://litllm.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑