Discrete diffusion models have recently emerged as a powerful class of generative models for chemistry and biology data. In these fields, the goal is to generate various samples with high rewards (e.g., drug-likeness in molecules), making reward-based guidance crucial. Most existing methods are based on guiding the diffusion model using intermediate rewards but tend to underperform since intermediate rewards are noisy due to the non-smooth nature of reward functions used in scientific domains. To address this, we propose Clean-Sample Markov Chain (CSMC) Sampler, a method that performs effective test-time reward-guided sampling for discrete diffusion models, enabling local search without relying on intermediate rewards. CSMC constructs a Markov chain of clean samples using the Metropolis-Hastings algorithm such that its stationary distribution is the target distribution. We design a proposal distribution by sequentially applying the forward and backward diffusion processes, making the acceptance probability tractable. Experiments on molecule and biological sequence generation with various reward functions demonstrate that our method consistently outperforms prior approaches that rely on intermediate rewards.
Aligning Few-Step Diffusion Models with Dense Reward Difference Learning
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（计算机学院、多媒体软件国家工程研究中心和多媒体与网络通信工程湖北省重点实验室、武汉大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（网络安全科学与技术学院、中山大学深圳校区）；TikTok, ByteDance（TikTok、字节跳动）；Tencent Inc.（腾讯公司）；College of Electronic and Information Engineering, Tongji University（电子信息工程学院、同济大学）；School of Medical Information and Engineering, Southwest Medical University（医学信息与工程学院、西南医科大学）；College of Computing and Data Science and the Generative AI Lab at Nanyang Technological University（计算与数据科学学院和南洋理工大学生成式AI实验室）
CellRFT: Reinforcement Fine-Tuning for Single-Cell Perturbation Modeling
Chinese Academy of Sciences（中国科学院）；Peking University（北京大学）；Renmin University of China（中国人民大学）；University of Chinese Academy of Sciences（中国科学院大学）；Central University of Finance and Economics（中央财经大学）
imitation provides open-source implementations of imitation and reward learning algorithms in PyTorch. We include three inverse reinforcement learning (IRL) algorithms, three imitation learning algorithms and a preference comparison algorithm. The implementations have been benchmarked against previous results, and automated tests cover 98% of the code. Moreover, the algorithms are implemented in a modular fashion, making it simple to develop novel algorithms in the framework. Our source code, including documentation and examples, is available at https://github.com/HumanCompatibleAI/imitation
The ability to accelerate the design of biological sequences can have a substantial impact on the progress of the medical field. The problem can be framed as a global optimization problem where the objective is an expensive black-box function such that we can query large batches restricted with a limitation of a low number of rounds. Bayesian Optimization is a principled method for tackling this problem. However, the astronomically large state space of biological sequences renders brute-force iterating over all possible sequences infeasible. In this paper, we propose MetaRLBO where we train an…
Biological Pathway Guided Gene Selection Through Collaborative Reinforcement Learning
University of Central Florida（佛罗里达中央大学）；University of Kansas（堪萨斯大学）；Vanderbilt University Medical Center（范德比尔特大学医学中心）；Arizona State University（亚利桑那州立大学）
A significant challenge in developing AI that can generalize well is designing agents that learn about their world without being told what to learn, and apply that learning to challenges with sparse rewards. Moreover, most traditional reinforcement learning approaches explicitly separate learning and decision making in a way that does not correspond to biological learning. We implement an architecture founded in principles of experimental neuroscience, by combining computationally efficient abstractions of biological algorithms. Our approach is inspired by research on spike-timing dependent pl…
We seek to (i) characterize the learning architectures exploited in biological neural networks for training on very few samples, and (ii) port these algorithmic structures to a machine learning context. The Moth Olfactory Network is among the simplest biological neural systems that can learn, and its architecture includes key structural elements and mechanisms widespread in biological neural nets, such as cascaded networks, competitive inhibition, high intrinsic noise, sparsity, reward mechanisms, and Hebbian plasticity. These structural biological elements, in combination, enable rapid learni…
