arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.21740cs.AI
收藏

SMDD-Bench: 大语言模型能否解决真实世界的小分子药物设计任务?

SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?

Carnegie Mellon University(卡内基梅隆大学);Stealth Pennsylvania State University(隐形宾夕法尼亚州立大学)

Kevin Han, Renfei Zhang, Kathy Wei, Hamed Mahdavi, Niloofar Mireshghallah, Amir Barati Farimani

AI总结:

提出SMDD-Bench基准,通过502个多轮长时任务实例评估LLM在真实小分子药物设计中的表现,发现最优模型GPT5.4仅解决40.2%任务。

摘要:

LLM智能体在科学发现应用中具有巨大潜力。然而,LLM智能体在跨不同化学空间和靶标的真实世界小分子药物设计(SMDD)任务上的表现尚不明确。当前的评估方法要么是临时的,对于真实发现过于简单,规模有限,或局限于单轮问答。为了标准化LLM智能体在小分子设计上的评估,我们引入了SMDD-Bench,一个具有挑战性的多轮长时智能体基准,包含502个保证可解的任务实例,涵盖5种任务类型:2D药效团识别、相互作用点发现、骨架跃迁、先导化合物优化和片段组装。SMDD-Bench任务覆盖广泛的化学空间,涉及102个独特的蛋白质靶标。完全解决该基准需要具备强大的化学和生物学推理能力及3D直觉,理解专业工具的使用,并在有限的oracle调用次数内展示规划专业知识。我们对7个前沿的开源和闭源LLM进行了基准测试,发现性能最好的LLM GPT5.4仅解决了40.2%的任务。我们希望SMDD-Bench能提供一个标准化的测试平台,激励该领域训练和评估用于全自动计算药物设计的LLM智能体。我们在smddbench.com上托管了一个公共排行榜。

英文摘要:

LLM agents have incredible potential for scientific discovery applications. However, the performance of LLM agents on real-world, small molecule drug design (SMDD) tasks across diverse chemistries and targets is unclear. Current evaluation methods are either ad hoc, too simple for real-world discovery, limited in scale, or restricted to single-turn question answering. In effort to standardize the evaluation of LLM agents on small molecule design, we introduce SMDD-Bench, a challenging, multi-turn, long-horizon agentic benchmark consisting of 502 guaranteed-solvable task instances spanning 5 task types: 2D Pharmacophore Identification, Interaction Point Discovery, Scaffold Hopping, Lead Optimization, and Fragment Assembly. SMDD-Bench tasks span a wide region of chemical space and involve 102 unique protein targets. Completely solving the benchmark would require having strong chemical and biological reasoning and 3D intuition, understanding specialized tool use, and displaying planning expertise over a limited number of oracle calls. We benchmark 7 frontier open and closed source LLMs and find even the most performant LLM, GPT5.4, solves only 40.2\% of tasks. We hope SMDD-Bench provides a standardized testbed to invigorate the field towards training and evaluating LLM agents for fully autonomous computational drug design. We host a public leaderboard at smddbench.com .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑