过度自信的预言机:计算机模拟序列设计基准测试的局限性
Overconfident Oracles: Limitations of In Silico Sequence Design Benchmarking
AI总结:
本研究考察12种利用机器学习预言机的序列设计方法,发现预言机差异导致基准结果不可靠,提出补充生物物理测量以提升稳健性。
摘要:
机器学习方法可以自动化生物序列的计算机模拟设计,旨在降低成本并加速医学研究。鉴于湿实验室的获取受限,计算机模拟设计方法通常使用预言机模型来评估从头生成的序列。然而,不同方法使用不同的预言机模型使得可靠比较变得困难,这引出了一个问题:计算机模拟序列设计基准是否可靠?在本工作中,我们考察了文献中常见的利用机器学习预言机的12种序列设计方法,发现它们在交叉一致性和可复现性方面存在显著挑战。实际上,架构不同甚至仅训练种子不同的预言机,被证明会产生相互矛盾的相对性能,我们的分析表明,分布外泛化能力差是一个关键问题。为应对这些挑战,我们建议在评估中补充一套生物物理测量指标,以评估生成序列的可行性并限制预言机需要评分的分布外序列,从而提高设计过程的稳健性。我们的工作旨在揭示当前评估过程中的潜在陷阱,并促进稳健基准的开发,最终推动计算机模拟设计方法的改进。
英文摘要:
Machine learning methods can automate the in silico design of biological sequences, aiming to reduce costs and accelerate medical research. Given the limited access to wet labs, in silico design methods commonly use an oracle model to evaluate de novo generated sequences. However, the use of different oracle models across methods makes it challenging to compare them reliably, motivating the question: are in silico sequence design benchmarks reliable? In this work, we examine 12 sequence design methods that utilise ML oracles common in the literature and find that there are significant challenges with their cross-consistency and reproducibility. Indeed, oracles differing by architecture, or even just training seed, are shown to yield conflicting relative performance with our analysis suggesting poor out-of-distribution generalisation as a key issue. To address these challenges, we propose supplementing the evaluation with a suite of biophysical measures to assess the viability of generated sequences and limit out-of-distribution sequences the oracle is required to score, thereby improving the robustness of the design procedure. Our work aims to highlight potential pitfalls in the current evaluation process and contribute to the development of robust benchmarks, ultimately driving the improvement of in silico design methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。