Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion
Huawei Technologies Co., Ltd.（华为技术有限公司）；Chinese Academy of Sciences（中国科学院）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）；Université de Montréal（蒙特利尔大学）
Weitao Du, Jiujiu Chen, Xuecang Zhang, Zhiming Ma, Shengchao Liu
Recently, artificial intelligence for drug discovery has raised increasing interest in both machine learning and chemistry domains. The fundamental building block for drug discovery is molecule geometry and thus, the molecule's geometrical representation is the main bottleneck to better utilize machine learning techniques for drug discovery. In this work, we propose a pretraining method for molecule joint auto-encoding (MoleculeJAE). MoleculeJAE can learn both the 2D bond (topology) and 3D conformation (geometry) information, and a diffusion process model is applied to mimic the augmented trajectories of such two modalities, based on which, MoleculeJAE will learn the inherent chemical structure in a self-supervised manner. Thus, the pretrained geometrical representation in MoleculeJAE is expected to benefit downstream geometry-related tasks. Empirically, MoleculeJAE proves its effectiveness by reaching state-of-the-art performance on 15 out of 20 tasks by comparing it with 12 competitive baselines.
Tokenization Tradeoffs in Structured EHR Foundation Models
Child Health Evaluative Sciences（儿童健康评估科学）；The Hospital for Sick Children（圣迈克尔医院）；University Health Network（大学健康网络）；Department of Medicine（医学部）；Division of Computational Medicine, Department of Medicine（计算医学科，医学部）；Stanford University（斯坦福大学）
Range-aware Positional Encoding via High-order Pretraining: Theory and Practice
FPT Software AI Center（FPT软件人工智能中心）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Computer Network Information Center, Chinese Academy of Sciences（中国科学院计算机网络信息中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）
Medical report generation is a challenging task since it is time-consuming and requires expertise from experienced radiologists. The goal of medical report generation is to accurately capture and describe the image findings. Previous works pretrain their visual encoding neural networks with large datasets in different domains, which cannot learn general visual representation in the specific medical domain. In this work, we propose a medical report generation framework that uses a contrastive learning approach to pretrain the visual encoder and requires no additional meta information. In additi…
It's All Connected: Topology-Aware Structural Graph Encoding Improves Performance on Polymer Prediction
Faculty of Engineering Science, University of Bayreuth, Germany（拜罗伊特大学工程科学学院）；Faculty of Mathematics, Physics & Computer Science, University of Bayreuth, Germany（拜罗伊特大学数学、物理与计算机科学学院）；Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Germany（耶拿弗里德里希·施莱尔大学有机与大分子化学实验室）；Jena Center for Soft Matter (JCSM), Friedrich Schiller University Jena, Germany（耶拿软物质中心（JCSM））；Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena), Germany（耶拿聚合物能源应用研究所（HIPOLE 耶拿））；Helmholtz Zentrum Berlin für Materialien und Energie GmbH (HZB), Germany（柏林材料与能源研究中心（HZB））
CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Kuaishou Technology（快手科技）
This brief paper reports the shortcoming of a variant of convolutional neural network whose components are developed based on the pattern theory framework.
Several recent results in machine learning have established formal connections between autoencoders---artificial neural network models that attempt to reproduce their inputs---and other coding models like sparse coding and K-means. This paper explores in depth an autoencoder model that is constructed using rectified linear activations on its hidden units. Our analysis builds on recent results to further unify the world of sparse linear coding models. We provide an intuitive interpretation of the behavior of these coding models and demonstrate this intuition using small, artificial datasets wit…
In this paper, we propose Orthogonal Generative Adversarial Networks (O-GANs). We decompose the network of discriminator orthogonally and add an extra loss into the objective of common GANs, which can enforce discriminator become an effective encoder. The same extra loss can be embedded into any kind of GANs and there is almost no increase in computation. Furthermore, we discuss the principle of our method, which is relative to the fully-exploiting of the remaining degrees of freedom of discriminator. As we know, our solution is the simplest approach to train a generative adversarial network w…
We started with a knowledge graph of connected entities and descriptive properties of those entities, from which, a hierarchical representation of the knowledge graph is derived. Using a graphical, energy-based neural network, we are able to show that the structure of the hierarchy can be internally captured by the neural network, which allows for efficient output of the underlying equilibrium distribution from which the data are drawn.
