测量2025年中期LLM对生物学新手表现的影响
Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology
AI总结:
2025年中期LLM对生物学新手在实验室任务中的表现影响有限,但显示出适度的性能优势。
摘要:
大型语言模型(LLMs)在生物基准测试中表现优异,这引发了担忧,认为它们可能帮助新手获得双重用途的实验室技能。然而,这种能力是否能提高人类在物理实验室中的表现仍不清楚。为此,我们进行了一项预注册、研究者盲法、随机对照试验(2025年6月至8月;n=153),评估LLMs是否能改善新手在模拟病毒逆转遗传学工作流程任务中的表现。我们观察到,工作流程完成的主要终点(5.2% LLM vs. 6.6%互联网;P=0.759)以及各个任务的成功率均无显著差异。然而,LLM组在四个任务中显示出数值更高的成功率,尤其是在细胞培养任务(68.8% LLM vs. 55.3%互联网;P=0.059)。对汇总数据的后验贝叶斯建模估计,在LLM协助下,一个
英文摘要:
Large language models (LLMs) perform strongly on biological benchmarks, raising concerns that they may help novice actors acquire dual-use laboratory skills. Yet, whether this translates to improved human performance in the physical laboratory remains unclear. To address this, we conducted a pre-registered, investigator-blinded, randomized controlled trial (June-August 2025; n = 153) evaluating whether LLMs improve novice performance in tasks that collectively model a viral reverse genetics workflow. We observed no significant difference in the primary endpoint of workflow completion (5.2% LLM vs. 6.6% Internet; P = 0.759), nor in the success rate of individual tasks. However, the LLM arm had numerically higher success rates in four of the five tasks, most notably for the cell culture task (68.8% LLM vs. 55.3% Internet; P = 0.059). Post-hoc Bayesian modeling of pooled data estimates an approximate 1.4-fold increase (95% CrI 0.74-2.62) in success for a "typical" reverse genetics task under LLM assistance. Ordinal regression modelling suggests that participants in the LLM arm were more likely to progress through intermediate steps across all tasks (posterior probability of a positive effect: 81%-96%). Overall, mid-2025 LLMs did not substantially increase novice completion of complex laboratory procedures but were associated with a modest performance benefit. These results reveal a gap between in silico benchmarks and real-world utility, underscoring the need for physical-world validation of AI biosecurity assessments as model capabilities and user proficiency evolve.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。