Large Language Models (LLMs) stand at the forefront of a number of Natural Language Processing (NLP) tasks. Despite the widespread adoption of LLMs in NLP, much of their potential in broader fields remains largely unexplored, and significant limitations persist in their design and implementation. Notably, LLMs struggle with structured data, such as graphs, and often falter when tasked with answering domain-specific questions requiring deep expertise, such as those in biology and chemistry. In this paper, we explore a fundamental question: Can LLMs effectively handle molecule prediction tasks? Rather than pursuing top-tier performance, our goal is to assess how LLMs can contribute to diverse molecule tasks. We identify several classification and regression prediction tasks across six standard molecule datasets. Subsequently, we carefully design a set of prompts to query LLMs on these tasks and compare their performance with existing Machine Learning (ML) models, which include text-based models and those specifically designed for analysing the geometric structure of molecules. Our investigation reveals several key insights: Firstly, LLMs generally lag behind ML models in achieving competitive performance on molecule tasks, particularly when compared to models adept at capturing the geometric structure of molecules, highlighting the constrained ability of LLMs to comprehend graph data. Secondly, LLMs show promise in enhancing the performance of ML models when used collaboratively. Lastly, we engage in a discourse regarding the challenges and promising avenues to harness LLMs for molecule prediction tasks. The code and models are available at https://github.com/zhiqiangzhongddu/LLMaMol.
SIU: A Million-Scale Structural Small Molecule-Protein Interaction Dataset for Unbiased Bioactivity Prediction
Peking University（北京大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；School of Life Sciences, Tsinghua University（清华大学生命科学学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）
AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（中国科学院计算技术研究所）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）
PredBench: Benchmarking Spatio-Temporal Prediction across Diverse Disciplines
Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Shanghai Jiao Tong University（上海交通大学）；Sydney University（悉尼大学）；The University of Hong Kong（香港大学）
New Benchmarking Shows Limited Generalization Power of TCR Antigenic Epitope Prediction Models
Trustworthy and Intelligent Computing Lab (TAIC), Department of Computer Science and Electrical Engineering, University of Maryland, Baltimore County（可信智能计算实验室（TAIC），计算机科学与电气工程系，马里兰大学巴尔的摩分校）；Children’s Hospital of Philadelphia（费城儿童医院）；Department of Bioengineering, University of Pennsylvania（生物工程系，宾夕法尼亚大学）；Institute for Immunology & Immune Health, University of Pennsylvania（免疫学与免疫健康研究所，宾夕法尼亚大学）；Institute for RNA Innovation, University of Pennsylvania（RNA创新研究所，宾夕法尼亚大学）；Abramson Cancer Center, University of Pennsylvania（Abramson癌症中心，宾夕法尼亚大学）；Center for Precision Engineering for Health, University of Pennsylvania（健康精准工程中心，宾夕法尼亚大学）；Center for Cellular Immunotherapies, University of Pennsylvania（细胞免疫治疗中心，宾夕法尼亚大学）
A Machine Learning Benchmarking Framework for Lipid Nanoparticle Transfection Efficiency Prediction
Department of Electrical and Computer Engineering, University of Southern California（电气与计算机工程系，南加州大学）；University of Cambridge（剑桥大学）；School of Pharmacy, University of Wisconsin-Madison（威斯康星大学麦迪逊分校药学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；East Tennessee State University（东田纳西州立大学）
ContextGNN goes to Elliot: Towards Benchmarking Relational Deep Learning for Static Link Prediction (aka Personalized Item Recommendation)
University of Barcelona（巴塞罗那大学）；University of Southern California（南加州大学）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；Inria（法国国家信息与自动化研究所）
Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis
Argonne National Laboratory（阿贡国家实验室）；Frederick National Laboratory for Cancer Research（弗雷德里克国家癌症实验室）；University of Nebraska–Lincoln（内布拉斯加大学林肯分校）；Texas Tech University（德克萨斯理工大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
Benchmarking Pathology Foundation Models for Breast Cancer Survival Prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Division of Oncology, Department of Clinical Sciences Lund, Lund University（卢德大学临床科学系肿瘤学系）；Oxford Suzhou Centre for Advanced Research, University of Oxford（牛津大学苏州先进研究中心）
Benchmarking Early Agitation Prediction in Community-Dwelling People with Dementia Using Multimodal Sensors and Machine Learning
KITE Research Institute（KITE研究机构）；Toronto Rehabilitation Institute（多伦多康复研究所）；University Health Network（大学健康网络）；Lawrence Bloomberg Faculty of Nursing（劳伦斯布罗德利护理学院）；University of Toronto（多伦多大学）；College of Engineering and Technology（工程与技术学院）；American University of the Middle East（中东美国大学）
Benchmarking non-conformity score functions in conformal prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet（卡罗林斯卡研究所医学流行病学与生物统计学系）；Department of Molecular Medicine and Surgery, Karolinska Institutet（卡罗林斯卡研究所分子医学与外科学系）
