Transformers for molecular property prediction: Domain adaptation efficiently improves performance
Spoken Language Systems, Saarland Informatics Campus, Saarland University（spoken language systems，萨尔兰信息学校园，萨尔兰大学）；Medical Faculty, Saarland University（医学学院，萨尔兰大学）；Center for Bioinformatics, Saarland University（生物信息学中心，萨尔兰大学）；Data Driven Drug Design, Center for Bioinformatics, Saarland University（数据驱动药物设计，生物信息学中心，萨尔兰大学）
Afnan Sultan, Max Rausch-Dupont, Shahrukh Khan, Olga Kalinina, Dietrich Klakow, Andrea Volkamer
Over the past six years, molecular transformer models have become key tools in drug discovery. Most existing models are pre-trained on large, unlabeled datasets such as ZINC or ChEMBL. However, the extent to which large-scale pre-training improves molecular property prediction remains unclear. This study evaluates transformer models for this task while addressing their limitations. We explore how pre-training dataset size and chemically informed objectives impact performance. Our results show that increasing the dataset beyond approximately 400K to 800K molecules from large-scale unlabeled databases does not enhance performance across seven datasets covering five ADME endpoints: lipophilicity, permeability, solubility (two datasets), microsomal stability (two datasets), and plasma protein binding. In contrast, domain adaptation on a small, domain-specific dataset (less than or equal 4K molecules) using multi-task regression of physicochemical properties significantly boosts performance (P-value less than 0.001). A model pre-trained on 400K molecules and adapted with domain-specific data outperforms larger models such as MolFormer and performs comparably to MolBERT. Benchmarks against Random Forest (RF) baselines using descriptors and Morgan fingerprints show that chemically and physically informed features consistently yield better performance across model types. While RF remains a strong baseline, we identify concrete practices to enhance transformer performance. Aligning pre-training and adaptation with chemically meaningful tasks and domain-relevant data presents a promising direction for molecular property prediction. Our models are available on HuggingFace for easy use and adaptation.
Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization
Dialpad Canada Inc.（Dialpad加拿大公司）；Information Retrieval & Knowledge Management Research Lab, York University（约克大学信息检索与知识管理研究实验室）；School of Information Technology, York University（约克大学信息技术学院）
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers
NEC Laboratories Europe GmbH（NEC欧洲实验室有限公司）；University of Mannheim（曼海姆大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）
Safe Self-Refinement for Transformer-based Domain Adaptation
Stony Brook University（石溪大学）；XPeng Motors（小鹏汽车）
无监督的领域自适应（UDA）旨在利用标签丰富的源域来解决相关未标记目标域上的任务。这是一个具有挑战性的问题，特别是当一个大的域之间的差距位于源和目标域。本文提出了一种新的解决方案SSRT（Safe Self-Refinement for Transformer-based领域自适应），从两个方面进行了改进。首先，由于视觉Transformer在各种视觉任务中的成功，我们为SSRT配备了一个Transformer骨干。我们发现，视觉Transformer与简单的对抗性适应的组合超过了最好的报告卷积神经网络（CNN）的结果，基于具有挑战性的DomainNet基准，显示其强大的可转移特征表示。其次，为了降低模型崩溃的风险，提高知识转移的有效性与大差距的领域之间，我们提出了一个安全的自我完善策略。具体而言，SSRT利用扰动的目标域数据的预测来细化模型。由于视觉Transformer的模型容量很大，并且在这种具有挑战性的任务中的预测可能是有噪声的，因此设计了一种安全的训练机制来自适应地调整学习配置。在几个广泛测试的UDA基准上进行了广泛的评估，SSRT始终获得最佳性能，包括在Bull-Home上的85.43%，在VisDA-2017上的88.76%和在DomainNet上的45.2%。
Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers
University of Bucharest（布加勒斯特大学）；Norwegian University of Science and Technology（挪威科技大学）；University of Medicine and Pharmacy “Carol Davila”（卡罗尔·达维拉医药大学）
PCaM: A Progressive Focus Attention-Based Information Fusion Method for Improving Vision Transformer Domain Adaptation
Westlake Institute for Advanced Studies（西湖研究学院）；Centre for Artificial Intelligence（人工智能中心）；State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation（自动化研究所）；Chinese Academy of Sciences (CASIA)（中国科学院）；School of Artificial Intelligence（人工智能学院）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Alibaba Group（阿里巴巴集团）；School of Computer Science（计算机学院）；Engineering, the Faculty of Innovation Engineering, Macau University of Science（工程学院，创新工程 faculty，澳门科学大学）；Microsoft（微软公司）
Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models
Northwestern University（西北大学）；University of Science and Technology of China（中国科学技术大学）；National Yang Ming Chiao Tung University（阳明交通大学）；Simons Institute UC Berkeley（伯克利Simons研究所）
Breaking Feedback-Blindness: Utility-Augmented Transformer for Sequential Decision Making
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of International Business and Economics（对外经济贸易大学）
Hyper-Decision Transformer for Efficient Online Policy Adaptation
Carnegie Mellon University（卡内基梅隆大学）；University of Montreal（蒙特利尔大学）；Mila（米拉人工智能研究所）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；UMass Amherst（马萨诸塞大学阿默斯特分校）
Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Shanghai Innovation Institute（上海创新研究院）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究所（TeleAI））；University of Science and Technology of China (USTC)（中国科学技术大学）
Freeze, Diffuse, Decode: Task-Aware Adaptation of Transformer Embeddings for Antimicrobial Peptide Design
Institute of AI for Health, Helmholtz Munich School of Computation, Information；Technology, Technical University of Munich Mila - Quebec AI institute；Univ. de Montreal Dept. Mathematics \& Statistics, Utah State University Department of Statistics, Brigham Young University Faculty of Mathematics, Informatics
