In the field of chemistry, the objective is to create novel molecules with desired properties, facilitating accurate property predictions for applications such as material design and drug screening. However, existing graph deep learning methods face limitations that curb their expressive power. To address this, we explore the integration of vast molecular domain knowledge from Large Language Models (LLMs) with the complementary strengths of Graph Neural Networks (GNNs) to enhance performance in property prediction tasks. We introduce a Multi-Modal Fusion (MMF) framework that synergistically harnesses the analytical prowess of GNNs and the linguistic generative and predictive abilities of LLMs, thereby improving accuracy and robustness in predicting molecular properties. Our framework combines the effectiveness of GNNs in modeling graph-structured data with the zero-shot and few-shot learning capabilities of LLMs, enabling improved predictions while reducing the risk of overfitting. Furthermore, our approach effectively addresses distributional shifts, a common challenge in real-world applications, and showcases the efficacy of learning cross-modal representations, surpassing state-of-the-art baselines on benchmark datasets for property prediction tasks.
LLMs Meet Cross-Modal Time Series Analytics: Overview and Directions
Nanyang Technological University（南洋理工大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科学与技术大学）；Technical University of Munich（慕尼黑技术大学）；King Abdullah University of Science and Technology（国王阿卜杜勒阿齐兹大学）
Multi-scale hypergraph meets LLMs: Aligning large language models for time series analysis
State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）；College of Computer Science and Technology, Zhejiang University（计算机科学与技术学院，浙江大学）
Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey
Zhejiang University（浙江大学）；Hangzhou Dianzi University（杭州电子科技大学）；The University of Manchester（曼彻斯特大学）；University of Oxford（牛津大学）；Southeast University（东南大学）；Purdue University（普渡大学）；The University of Edinburgh（爱丁堡大学）
Advancing Enterprise Spatio-Temporal Forecasting Applications: Data Mining Meets Instruction Tuning of Language Models For Multi-modal Time Series Analysis in Low-Resource Settings
Vision-Language Meets the Skeleton: Progressively Distillation with Cross-Modal Knowledge for 3D Action Representation Learning
School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）；The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Polytechnic University Shenzhen Research Institute（香港理工大学深圳研究院）；No.1 Orthopedic Hospital of Chengdu（成都市第一骨科医院）
From Words to Molecules: A Survey of Large Language Models in Chemistry
School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；City University of Hong Kong（香港城市大学）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Probing the limitations of multimodal language models for chemistry and materials research
Friedrich Schiller University Jena（弗里德里希·席勒大学耶拿分校）；Indian Institute of Technology Delhi（印度理工学院德里分校）；Center for Energy and Environmental Chemistry Jena (CEEC Jena)（耶拿能源与环境化学中心）；Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena)（耶拿亥姆霍兹能源应用聚合物研究所）；Jena Center for Soft Matter (JCSM)（耶拿软物质中心）
Stanford University（斯坦福大学）；Intel AI Lab（英特尔人工智能实验室）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）
A Review of Large Language Models and Autonomous Agents in Chemistry
FutureHouse Inc.（FutureHouse公司）；University of Rochester（罗切斯特大学）；School of Chemistry and Materials Science（化学与材料科学学院）；Rochester Institute of Technology（罗切斯特理工学院）
A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language
Renmin University of China（中国人民大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；The Chinese University of Hong Kong（香港中文大学）
