Recently, text-to-molecule models have shown great potential across various chemical applications, e.g., drug-discovery. These models adapt language models to molecular data by representing molecules as sequences of atoms. However, they rely on atom-level tokenizations, which primarily focus on modeling local connectivity, thereby limiting the ability of models to capture the global structural context within molecules. To tackle this issue, we propose a novel text-to-molecule model, coined Context-Aware Molecular T5 (CAMT5). Inspired by the significance of the substructure-level contexts in understanding molecule structures, e.g., ring systems, we introduce substructure-level tokenization for text-to-molecule models. Building on our tokenization scheme, we develop an importance-based training strategy that prioritizes key substructures, enabling CAMT5 to better capture the molecular semantics. Extensive experiments verify the superiority of CAMT5 in various text-to-molecule generation tasks. Intriguingly, we find that CAMT5 outperforms the state-of-the-art methods using only 2% of training tokens. In addition, we propose a simple yet effective ensemble strategy that aggregates the outputs of text-to-molecule models to further boost the generation performance. Code is available at https://github.com/Songhyeontae/CAMT5.git.
HIGHT: Hierarchical Graph Tokenization for Molecule-Language Alignment
MBZUAI(穆罕默德·本·扎耶德人工智能大学);Carnegie Mellon University(卡内基梅隆大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);University of Maryland, College Park(马里兰大学学院市分校);National University of Singapore(新加坡国立大学)
Molecule AI(分子人工智能公司);Manning College of Information and Computer Sciences, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校曼宁信息与计算机科学学院);Happening Technology Ltd(Happening科技有限公司);Indian Institute of Technology Delhi(印度理工学院德里分校)
Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization
Sound and Music Computing Lab, School of Computing, NUS(新加坡国立大学计算机学院声音与音乐计算实验室);Courant Institute of Mathematical Sciences, New York University(纽约大学应用数学科学学院);Music X Lab, MBZUAI(MBZUAI音乐X实验室)
MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Technical University of Berlin(柏林工业大学)