arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.03057cs.LGcs.AI
收藏

重新思考用于大语言模型的分子文本表示:一项实证研究

Rethinking Molecular Text Representations for LLMs: An Empirical Study

University of Oxford(牛津大学);DSO National Laboratories(DSO国家实验室)

Arun Raja, Garrett M. Morris, Kian Ming A. Chai

AI总结:

通过系统基准测试,评估了9种分子表示和8种化学任务下16个LLM的性能,发现表示选择强烈影响结果,结构化文本表示(CML、MolJSON)在结构任务中占优,IUPAC在语义任务中占优,而SMILES很少最优。

摘要:

大语言模型(LLMs)越来越多地用于分子任务,但目前尚不清楚使用哪种分子表示。我们提出了一个系统基准测试,评估了LLM在九种表示和八种化学任务上的分子能力。我们基准测试了16个LLM,涵盖五个模型家族,包括推理和非推理变体、化学专用LLM以及封闭前沿模型。性能强烈依赖于表示,没有单一表示在所有任务中获胜,尽管CML是最好的,其次是MolJSON、InChI,然后是规范SMILES。显式结构化文本表示(CML和MolJSON)主导结构任务;IUPAC主导语义任务,在所有16个LLM的分子检索中获胜;而SMILES变体尽管在预训练中普遍存在,但很少是最优的。化学专用模型在使用SMILES时表现良好,但使用结构化文本表示时性能大幅下降,这表明仅基于SMILES的评估奖励了不具泛化能力的专业化。使用LLM作为评判者,我们发现IUPAC产生的正确分子生成比例最高。通过分词审计、线性探针和注意力的机制研究表明,表示在模型内部以不同方式编码;例如,结构化表示需要跨分子范围的更高注意力。我们的结果反对表示不变的评估,并激励基于LLM的化学任务感知表示路由。

英文摘要:

Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑