AutoBool: 一种基于强化学习训练的LLM,用于系统综述中高效的布尔查询生成
AutoBool: An Reinforcement-Learning trained LLM for Effective Automated Boolean Query Generation for Systematic Reviews
AI总结:
AutoBool通过强化学习训练LLM生成高效的布尔查询,显著优于传统方法并接近专家水平。
摘要:
我们介绍了AutoBool,一种基于强化学习(RL)的框架,用于训练大型语言模型(LLMs)生成有效的布尔查询用于医学系统综述。布尔查询是该领域文献检索的主要机制,必须在高召回率的同时保持合理的精度——这一平衡往往使现有的基于提示的LLM方法难以实现。该领域的一个主要限制是缺乏高质量的布尔查询地面真相,这使得监督微调不切实际。AutoBool通过使用强化学习直接优化查询生成,而无需目标查询来解决这一挑战。为了支持这一努力,我们创建并发布了迄今为止最大的此类数据集:总计65588个主题用于训练和评估自动布尔查询生成任务。在我们新数据集和两个已建立的数据集(CLEF TAR和Seed Collection)上的实验表明,AutoBool显著优于零样本/少样本提示方法,并且在使用较小的模型架构时,其效果可与更大的基于GPT的模型(如GPT-4o、O3)相媲美。它还接近专家撰写的查询效果,同时检索的文档数量减少10至16倍。消融研究揭示了模型架构、大小、解码温度和提示设计的关键作用。代码和数据可在https://github.com/ielab/AutoBool上获得。
英文摘要:
We present AutoBool, a reinforcement learning (RL) framework that trains large language models (LLMs) to generate effective Boolean queries for medical systematic reviews. Boolean queries are the primary mechanism for literature retrieval in this domain and must achieve high recall while maintaining reasonable precision - a challenging balance that existing prompt-based LLM approaches often struggle to achieve. A major limitation in this space is the lack of high-quality ground-truth Boolean queries for each topic, which makes supervised fine-tuning impractical. AutoBool addresses this challenge by using RL to directly optimize query generation with retrieval measures, without requiring target queries. To support this effort, we create and release the largest dataset of its kind: 65588 topics in total for training and evaluating the task of automatic Boolean query formulation. Experiments on our new dataset and two established datasets (CLEF TAR and Seed Collection) show that AutoBool significantly outperforms zero shot/few shot prompting and matches or exceeds the effectiveness of much larger GPT-based models (e.g., GPT-4o, O3) using smaller backbones. It also approaches effectiveness of expert-authored queries while retrieving 10 to 16 times fewer documents. Ablation studies reveal the critical roles of model backbone, size, decoding temperature, and prompt design. Code and data are available at https://github.com/ielab/AutoBool.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。