Large Language Models (LLMs) with their strong task-handling capabilities have shown remarkable advancements across a spectrum of fields, moving beyond natural language understanding. However, their proficiency within the chemistry domain remains restricted, especially in solving molecule-related tasks. This challenge is attributed to their inherent limitations in comprehending molecules using only common textual representations, i.e. SMILES strings. In this study, we seek to enhance the ability of LLMs to comprehend molecules by equipping them with a multi-modal external module, termed MolX. Instead of directly using SMILES strings to represent a molecule, we utilize specific encoders to extract fine-grained features from both SMILES string and 2D molecular graph representations for feeding into an LLM. A hand-crafted molecular fingerprint is incorporated to leverage its embedded domain knowledge. To establish an alignment between MolX and the LLM's textual input space, the model in which the LLM is frozen, is pre-trained with a strategy including a diverse set of tasks. Experimental evaluations show that our proposed method outperforms baselines across downstream molecule-related tasks ranging from molecule-to-text translation to molecular property prediction, with and without fine-tuning the LLM, while only introducing a small number of trainable parameters-0.53% and 0.82%, respectively.
Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionality
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学,教育部脑启发智能感知与认知重点实验室);Independent Researcher(独立研究员)
IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes
School of AIA, Huazhong University of Science and Technology(华中科技大学人工智能学院);Deepeleph Intelligent Technology(DeepEleph智能技术);JD Explore Academy(JD探索学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)
Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration
Department of Electronics;Information \ 'an Jiaotong University Xi'an China;School of Information;Communications Engineering \ 'an Jiaotong University Xi'an China;School of Mathematics;Statistics \ 'an Jiaotong University Xi'an China;Information \ 'an Jiaotong University;Communications Engineering \ 'an Jiaotong University;Statistics \ 'an Jiaotong University
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
University of Verona(维罗纳大学);Sapienza University of Rome(罗马大学);University of Trento(特伦托大学);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);Reykjavik University(雷克雅未克大学)
Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data
Electrical and Computer Engineering, Queen’s University(皇后大学电气与计算机工程学院);Mechanical and Materials Engineering, Queen’s University(皇后大学机械与材料工程学院);Ingenuity Labs Research Institute, Queen’s University(皇后大学创新实验室研究 institute)
ProcessThinker: Enhancing Multi-modal Large Language Models Reasoning via Rollout-based Process Reward
LMU Munich(慕尼黑大学);Harvard University(哈佛大学);University of Cambridge(剑桥大学);Mina AI;Konrad Zuse School of Excellence in Reliable AI (relAI)(康拉德·楚泽可靠人工智能卓越学校(relAI))
Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
Department of Computer Science, University of Kaiserslautern-Landau (RPTU)(科斯拉尔特伦大学计算机科学系);SDS, German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));INRAE, UMR TETIS, University of Montpellier(蒙彼利埃大学UMR TETIS);CIRAD, UMR TETIS, University of Montpellier(蒙彼利埃大学UMR TETIS);INRIA, EVERGREEN, University of Montpellier(蒙彼利埃大学)
Enhancing Osteoporosis Detection: An Explainable Multi-Modal Learning Framework with Feature Fusion and Variable Clustering
School of Electrical and Computer Engineering, College of Engineering, University of Tehran(塔里班大学电气与计算机工程学院);Department of Epidemiology, Shiraz University of Medical Science(谢尔兹医学科学大学流行病学系);Department of Computer Science and Engineering, Sejong University(世宗大学计算机科学与工程系);School of Computing and Information Science, Anglia Ruskin University(安格利亚 Ruskin 大学计算与信息科学学院)
Enhancing Cross-Modal Medical Image Segmentation through Compositionality
University of Amsterdam(阿姆斯特丹大学);The Netherlands Cancer Institute(荷兰癌症研究所);Utrecht University(乌得勒支大学);Maastricht University Medical Center(马斯特里赫特大学医学中心)
本文提出一种端到端跨模态医学图像分割框架,通过引入可学习的 von Mises-Fisher 核实现组合性归纳偏置与内容-风格解耦,有效提升了分割性能、可解释性并降低计算成本。
Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency
Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);School of Information Management, Jiangxi University of Finance and Economics(江西财经大学信息管理学院);Institute for Infocomm Research, A*STAR(新加坡科技研究局信息通信研究所);School of Biomedical Engineering, Shenzhen University(深圳大学生物医学工程学院)
ELMAR: Enhancing LiDAR Detection with 4D Radar Motion Awareness and Cross-modal Uncertainty
Infineon Technologies AG and the Technical University of Munich(英飞凌科技有限公司和技术大学慕尼黑);China University of Geosciences(中国地质大学);Technical University of Munich(技术大学慕尼黑)
EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications(智能工程与自动化学院,北京邮电大学);State Grid Corporation of China(国家电网公司);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(人工智能学院,北京邮电大学)
Perceive and Calibrate: Analyzing and Enhancing Robustness of Medical Multi-Modal Large Language Models
Department of Computer Science and Engineering, CUHK, Hong Kong, China(计算机科学与工程系,CUHK,香港,中国);Institute of Medical Intelligence and XR, CUHK, Hong Kong, China(医学智能与XR研究所,CUHK,香港,中国)
Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);Center for Long-term Artificial Intelligence(长期人工智能中心);Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Chinese Academy of Sciences(中国科学院脑认知与脑智能技术重点实验室)
Enhancing CT Image synthesis from multi-modal MRI data based on a multi-task neural network framework
Kennedy Krieger Institute(肯尼迪克里格研究所);Columbia University(哥伦比亚大学);Johns Hopkins University School of Medicine(约翰斯·霍普金斯大学医学院);Johns Hopkins University(约翰斯·霍普金斯大学)
Enhancing Multimodal Medical Image Classification using Cross-Graph Modal Contrastive Learning
Department of Systems Engineering, Stevens Institute of Technology, Hoboken, New Jersey, USA(系统工程系,史蒂文斯理工学院);Department of Nuclear Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan(高雄长庚纪念医院核医学部);Institute of Statistics, National University of Kaohsiung, Kaohsiung, Taiwan(国立高雄大学统计研究所);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China(深圳先进技术研究院,中国科学院)