ROBoto2:一个用于LLM辅助临床试验偏倚风险评估的交互式系统与数据集
ROBoto2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment
AI总结:
ROBoto2是一个开源Web平台,通过结合PDF解析、检索增强LLM提示和人机协同审查,简化临床试验偏倚风险评估,并发布包含521份报告的数据集,评估4个LLM性能,推动自动化系统评价研究。
摘要:
我们提出了ROBOTO2,一个开源的、基于Web的平台,用于大型语言模型(LLM)辅助的临床试验偏倚风险(ROB)评估。ROBOTO2通过一个结合PDF解析、检索增强LLM提示和人机协同审查的交互式界面,简化了传统上劳动密集型的ROB v2(ROB2)标注过程。用户可以上传临床试验报告,接收ROB2信号问题的初步答案和支持证据,并对系统建议提供实时反馈或更正。ROBOTO2已在https://roboto2.vercel.app/公开提供,代码和数据已发布以促进可复现性和采用。我们构建并发布了一个包含521份儿科临床试验报告的数据集(8954个信号问题和1202个证据段落),使用手动和LLM辅助方法进行标注,作为基准并支持未来研究。使用该数据集,我们评估了4个LLM的ROB2性能,并分析了当前模型能力及自动化系统评价这一关键方面的持续挑战。
英文摘要:
We present ROBOTO2, an open-source, web-based platform for large language model (LLM)-assisted risk of bias (ROB) assessment of clinical trials. ROBOTO2 streamlines the traditionally labor-intensive ROB v2 (ROB2) annotation process via an interactive interface that combines PDF parsing, retrieval-augmented LLM prompting, and human-in-the-loop review. Users can upload clinical trial reports, receive preliminary answers and supporting evidence for ROB2 signaling questions, and provide real-time feedback or corrections to system suggestions. ROBOTO2 is publicly available at https://roboto2.vercel.app/, with code and data released to foster reproducibility and adoption. We construct and release a dataset of 521 pediatric clinical trial reports (8954 signaling questions with 1202 evidence passages), annotated using both manually and LLM-assisted methods, serving as a benchmark and enabling future research. Using this dataset, we benchmark ROB2 performance for 4 LLMs and provide an analysis into current model capabilities and ongoing challenges in automating this critical aspect of systematic review.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。