Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a systematic benchmark, MolRepBench, to evaluate LLM molecular competence across nine representations and eight chemical tasks. We benchmark 16 LLMs across five model families, including reasoning and non-reasoning variants, chemistry-specialized LLMs, and closed frontier models. Performance is strongly representation-dependent and no single representation wins across tasks, though CML is the best, followed by MolJSON, InChI and canonical SMILES. Explicit structured text representations (CML and MolJSON) dominate structural tasks; IUPAC dominates semantic tasks, winning molecule retrieval for all 16 LLMs; and SMILES variants are rarely optimal despite their prevalence in pretraining. Chemistry-specialized models perform well with SMILES at the cost of large degradations with structured text representations, suggesting SMILES-only evaluation rewards specialization that does not generalize. Using LLM-as-a- judge, we find that IUPAC produces the highest fraction of correct molecule generations. A mechanistic study via tokenization audits, linear probes, and attention shows that representations are encoded differently inside the model; for example, structured representations require more attention across the molecular span. Our results argue against representation-invariant evaluation and motivate task-aware representation routing for LLM-based chemistry.
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany(人工智能与社会、科学与技术部门,柏林Zuse研究所);Institute of Mathematics, Technische Universität Berlin, Germany(数学研究所,柏林技术大学)
LoReC: Rethinking Large Language Models for Graph Data Analysis
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tongji University(同济大学);Westlake University(西湖大学);Ant Group(蚂蚁集团);The Hong Kong University of Science and Technology(香港科技大学)
Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs
State Key Lab of Processors, Institute of Computing Technology, CAS(中国科学院计算技术研究所状态关键实验室);University of Chinese Academy of Sciences(中国科学院大学);People’s Public Security University of China(中国人民公安大学);Chinese University of Hong Kong(香港大学)
Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
Institute of Engineering Research, Korea University(韩国大学工程研究所);Kim Jaechul Graduate School of AI, KAIST(韩国科学技术院金在哲人工智能研究生院);Industrial Management Engineering, Korea University(韩国大学产业管理工程学院)
Rethinking Prospect Theory for LLMs: Revealing the Instability of Decision-Making under Epistemic Uncertainty
Hong Kong University of Science and Technology(香港科技大学);Huazhong University of Science and Technology(华中科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Fudan University, Shanghai, China(复旦大学);University of Bath, Bath, United Kingdom(巴斯大学);Oxford Suzhou Centre for Advanced Research, Suzhou, China(牛津苏格兰先进研究中心);Shanghai Innovation Institute, Shanghai, China(上海创新研究院);Department of Computer Science, University of Colorado Boulder, Colorado, USA(计算机科学系,科罗拉多大学博尔德分校);Department of Electrical Engineering and Computer Science, University of Michigan(电气工程与计算机科学系,密歇根大学)
Flowco: Rethinking Data Analysis in the Age of LLMs
Williams College(威廉姆斯学院);University of California, Los Angeles(加州大学洛杉矶分校);University of Massachusetts Amherst / Amazon Web Services(马萨诸塞大学阿默斯特分校 / 亚马逊网络服务)
The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));National University of Singapore(新加坡国立大学)
Rethinking the Understanding Ability across LLMs through Mutual Information
Department of Industrial and Systems Engineering(工业与系统工程系);University of Houston(休斯敦大学);Bakar Computational Health Sciences Institute(巴卡计算健康科学研究所);UCSF
Rethinking Repetition Problems of LLMs in Code Generation
Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education(高可信软件技术教育部重点实验室(北京大学));School of Computer Science, Peking University(北京大学计算机学院)
ToLeaP: Rethinking Development of Tool Learning with Large Language Models
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Dalian University of Technology(大连理工大学);University of Sydney(悉尼大学);Waseda University(早稻田大学);National University of Singapore(新加坡国立大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院)
Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models
Virginia Tech(弗吉尼亚理工大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校);University of California, Davis(加州大学戴维斯分校)