Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks. These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right. This paper studies the influence of token generation order on model quality via a novel two-pass language model that produces partially-filled sentence "templates" and then fills in missing tokens. We compare various strategies for structuring these two passes and observe a surprisingly large variation in model quality. We find the most effective strategy generates function words in the first pass followed by content words in the second. We believe these experimental results justify a more extensive investigation of generation order for neural language models.
University of Amsterdam（阿姆斯特丹大学）；Leiden University（莱顿大学）
Recent work has shown that recurrent neural networks (RNNs) can implicitly capture and exploit hierarchical information when trained to solve common natural language processing tasks such as language modeling (Linzen et al., 2016) and neural machine translation (Shi et al., 2016). In contrast, the ability to model structured data with non-recurrent neural networks has received little attention despite their success in many NLP tasks (Gehring et al., 2017; Vaswani et al., 2017). In this work, we compare the two architectures---recurrent versus non-recurrent---with respect to their ability to mo…
We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine translation (En-De and En-Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
Yandex（Yandex（亚德斯））；National Research University Higher School of Economics（国立研究型高等经济大学）；University of Amsterdam（阿姆斯特丹大学）
The dominant approach to sequence generation is to produce a sequence in some predefined order, e.g. left to right. In contrast, we propose a more general model that can generate the output sequence by inserting tokens in any arbitrary order. Our model learns decoding order as a result of its training procedure. Our experiments show that this model is superior to fixed order models on a number of sequence generation tasks, such as Machine Translation, Image-to-LaTeX and Image Captioning.
Conditional Feature Importance with Generative Modeling Using Adversarial Random Forests
Leibniz Institute for Prevention Research & Epidemiology – BIPS（莱布尼茨预防研究与流行病学研究所 – BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不来梅大学数学与计算机科学学院）；Berlin School of Economics and Law（柏林经济与法律学院）；University of Copenhagen（哥本哈根大学）
New York University（纽约大学）；Facebook AI Research（Facebook人工智能研究院）
We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher…
The last advances in sequence modeling are mainly based on deep learning approaches. The current state of the art involves the use of variations of the standard LSTM architecture, combined with several tricks that improve the final prediction rates of the trained neural networks. However, in some cases, these adaptations might be too much tuned to the particular problems being addressed. In this article, we show that a very simple idea, to add a direct connection between the input and the output, skipping the recurrent module, leads to an increase of the prediction accuracy in sequence modelin…
Multi-fidelity methods leverage low-cost surrogate models to speed up computations and make occasional recourse to expensive high-fidelity models to establish accuracy guarantees. Because surrogate and high-fidelity models are used together, poor predictions by surrogate models can be compensated with frequent recourse to high-fidelity models. Thus, there is a trade-off between investing computational resources to improve the accuracy of surrogate models versus simply making more frequent recourse to expensive high-fidelity models; however, this trade-off is ignored by traditional modeling met…
Alleviating the delayed feedback problem is of crucial importance for the conversion rate(CVR) prediction in online advertising. Previous delayed feedback modeling methods using an observation window to balance the trade-off between waiting for accurate labels and consuming fresh feedback. Moreover, to estimate CVR upon the freshly observed but biased distribution with fake negatives, the importance sampling is widely used to reduce the distribution bias. While effective, we argue that previous approaches falsely treat fake negative samples as real negative during the importance weighting and …
University of Cambridge（剑桥大学）；Microsoft Research Cambridge（微软剑桥研究院）；Tehran Institute for Advanced Studies（德黑兰高等研究院）
Variational Autoencoders (VAEs) are known to suffer from learning uninformative latent representation of the input due to issues such as approximated posterior collapse, or entanglement of the latent space. We impose an explicit constraint on the Kullback-Leibler (KL) divergence term inside the VAE objective function. While the explicit constraint naturally avoids posterior collapse, we use it to further understand the significance of the KL term in controlling the information transmitted through the VAE channel. Within this framework, we explore different properties of the estimated posterior…
Aplicaciones en Informática Avanzada (AIA)（高级信息应用公司）；University of Geneva（日内瓦大学）
The generation of accurate neutrino-nucleus cross-section models needed for neutrino oscillation experiments require simultaneously the description of many degrees of freedom and precise calculations to model nuclear responses. The detailed calculation of complete models makes the Monte Carlo generators slow and impractical. We present Exhaustive Neural Importance Sampling (ENIS), a method based on normalizing flows to find a suitable proposal density for rejection sampling automatically and efficiently, and discuss how this technique solves common issues of the rejection algorithm.
LIPN（巴黎第十三大学计算机科学实验室）；Université Sorbonne Paris Nord（索邦巴黎北大学）；INRIA Paris（法国国家信息与自动化研究所巴黎分部）
Even though Variational Autoencoders (VAEs) are widely used for semi-supervised learning, the reason why they work remains unclear. In fact, the addition of the unsupervised objective is most often vaguely described as a regularization. The strength of this regularization is controlled by down-weighting the objective on the unlabeled part of the training set. Through an analysis of the objective of semi-supervised VAEs, we observe that they use the posterior of the learned generative model to guide the inference model in learning the partially observed latent variable. We show that given this …
News headline generation is an essential problem of text summarization because it is constrained, well-defined, and is still hard to solve. Models with a limited vocabulary can not solve it well, as new named entities can appear regularly in the news and these entities often should be in the headline. News articles in morphologically rich languages such as Russian require model modifications due to a large number of possible word forms. This study aims to validate that models with a possibility of copying words from the original article performs better than models without such an option. The p…
Psychiatric neuroscience is increasingly aware of the need to define psychopathology in terms of abnormal neural computation. The central tool in this endeavour is the fitting of computational models to behavioural data. The most prominent example of this procedure is fitting reinforcement learning (RL) models to decision-making data collected from mentally ill and healthy subject populations. These models are generative models of the decision-making data themselves, and the parameters we seek to infer can be psychologically and neurobiologically meaningful. Currently, the gold standard approa…
Training datasets for machine learning often have some form of missingness. For example, to learn a model for deciding whom to give a loan, the available training data includes individuals who were given a loan in the past, but not those who were not. This missingness, if ignored, nullifies any fairness guarantee of the training procedure when the model is deployed. Using causal graphs, we characterize the missingness mechanisms in different real-world scenarios. We show conditions under which various distributions, used in popular fairness algorithms, can or can not be recovered from the trai…
Duke University（杜克大学）；University at Buffalo（布法罗大学）；Microsoft Dynamics 365 AI（微软Dynamics 365 AI）；Citadel LLC（城堡投资有限责任公司）；DeepMind（深度思维）
Auto-regressive text generation models usually focus on local fluency, and may cause inconsistent semantic meaning in long text generation. Further, automatically generating words with similar semantics is challenging, and hand-crafted linguistic rules are difficult to apply. We consider a text planning scheme and present a model-based imitation-learning approach to alleviate the aforementioned issues. Specifically, we propose a novel guider network to focus on the generative process over a longer horizon, which can assist next-word prediction and provide intermediate rewards for generator opt…
Neural random fields (NRFs), referring to a class of generative models that use neural networks to implement potential functions in random fields (a.k.a. energy-based models), are not new but receive less attention with slow progress. Different from various directed graphical models such as generative adversarial networks (GANs), NRFs provide an interesting family of undirected graphical models for generative modeling. In this paper we propose a new approach, the inclusive-NRF approach, to learning NRFs for continuous data (e.g. images), by introducing inclusive-divergence minimized auxiliary …
We investigate the problem of modeling symbolic sequences of polyphonic music in a completely general piano-roll representation. We introduce a probabilistic model based on distribution estimators conditioned on a recurrent neural network that is able to discover temporal dependencies in high-dimensional sequences. Our approach outperforms many traditional models of polyphonic music on a variety of realistic datasets. We show how our musical language model can serve as a symbolic prior to improve the accuracy of polyphonic transcription.
Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ~280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate wit…
Ludwig Maximilian University（路德维希-马克西米利安-慕尼黑大学）；Siemens AG Corporate Technology（西门子股份公司企业技术部）
With the rising number of interconnected devices and sensors, modeling distributed sensor networks is of increasing interest. Recurrent neural networks (RNN) are considered particularly well suited for modeling sensory and streaming data. When predicting future behavior, incorporating information from neighboring sensor stations is often beneficial. We propose a new RNN based architecture for context specific information fusion across multiple spatially distributed sensor stations. Hereby, latent representations of multiple local models, each modeling one sensor station, are jointed and weight…
FitCF: A Framework for Automatic Feature Importance-guided Counterfactual Example Generation
Quality and Usability Lab, Technische Universität Berlin（柏林技术大学质量与可用性实验室）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Deutsche Telekom（德国电信）；Saarland Informatics Campus（萨尔州信息学校区）；Centre for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）
Swisscom（瑞士电信）；Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Symphony
Recommender systems research tends to evaluate model performance offline and on randomly sampled targets, yet the same systems are later used to predict user behavior sequentially from a fixed point in time. Simulating online recommender system performance is notoriously difficult and the discrepancy between online and offline behaviors is typically not accounted for in offline evaluations. This disparity permits weaknesses to go unnoticed until the model is deployed in a production setting. In this paper, we first demonstrate how omitting temporal context when evaluating recommender system pe…
Beijing National Research Center for Information Science and Technology（北京国家信息科学与技术研究中心）；Tsinghua University（清华大学）
In terms of Generative Adversarial Networks (GANs), the information metric to discriminate the generative data from the real data, lies in the key point of generation efficiency, which plays an important role in GAN-based applications, especially in anomaly detection. As for the original GAN, there exist drawbacks for its hidden information measure based on KL divergence on rare events generation and training performance for adversarial networks. Therefore, it is significant to investigate the metrics used in GANs to improve the generation ability as well as bring gains in the training process…
With the increasing scale, complexity, and heterogeneity of the next generation networked systems, seamless control, management, and security of such systems becomes increasingly challenging. Many diverse applications have driven interest in networked systems, including large-scale distributed learning, multi-agent optimization, 5G service provisioning, and network slicing, etc. In this paper, we propose trust as a measure to evaluate the status of network agents and improve the decision-making process. We interpret trust as a relation among entities that participate in various protocols. Trus…
On the Importance of Step-wise Embeddings for Heterogeneous Clinical Time-Series
ETH Zürich（苏黎世联邦理工学院）；ETH AI Center（ETH人工智能中心）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Zürich University Hospital（苏黎世大学医院）；Swiss Institute of Bioinformatics（瑞士生物信息学研究所）；Department of Biology, ETH Zürich（苏黎世联邦理工学院生物学系）
Deep Learning (DL) techniques for Natural Language Processing have been evolving remarkably fast. Recently, the DL advances in language modeling, machine translation and paragraph understanding are so prominent that the potential of DL in Software Engineering cannot be overlooked, especially in the field of program learning. To facilitate further research and applications of DL in this field, we provide a comprehensive review to categorize and investigate existing DL methods for source code modeling and generation. To address the limitations of the traditional source code models, we formulate …
Multimodal Crystal Flow: Any-to-Any Modality Generation for Unified Crystal Modeling
Graduate School of AI, KAIST, Seoul, South Korea（韩国科学技术院人工智能研究生院，首尔，韩国）；Materials Intelligence Lab, LG AI Research, Seoul, South Korea（LG AI研究所材料智能实验室，首尔，韩国）
Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity
Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA（语言与语音处理中心（CLSP），约翰霍普金斯大学，美国）；Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, USA（稳健语音系统中心（CRSS），德克萨斯大学达拉斯分校，美国）；Language Technologies Institute (LTI), Carnegie Mellon University, USA（语言技术研究所（LTI），卡内基梅隆大学，美国）
NExT-Mol: 3D Diffusion Meets 1D Language Modeling for 3D Molecule Generation
National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；Chinese University of Hong Kong（香港中文大学）；Hokkaido University（北海道大学）
Carnegie Mellon University（卡内基梅隆大学）；Meta Reality Lab（Meta现实实验室）；University of Toronto（多伦多大学）
Understanding and modeling human behavior is fundamental to almost any computer vision and robotics applications that involve humans. In this thesis, we take a holistic approach to human behavior modeling and tackle its three essential aspects -- simulation, perception, and generation. Throughout the thesis, we show how the three aspects are deeply connected and how utilizing and improving one aspect can greatly benefit the other aspects. We also discuss the lessons learned and our vision for what is next for human behavior modeling.
Birla Institute of Technology and Science, Pilani（皮拉尼比尔拉理工学院）
In this paper, we propose a generic technique to model temporal dependencies and sequences using a combination of a recurrent neural network and a Deep Belief Network. Our technique, RNN-DBN, is an amalgamation of the memory state of the RNN that allows it to provide temporal information and a multi-layer DBN that helps in high level representation of the data. This makes RNN-DBNs ideal for sequence generation. Further, the use of a DBN in conjunction with the RNN makes this model capable of significantly more complex data representation than an RBM. We apply this technique to the task of poly…
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
In this paper, we proposed and validated a fully automatic pipeline for hippocampal surface generation via 3D U-net coupled with active shape modeling (ASM). Principally, the proposed pipeline consisted of three steps. In the beginning, for each magnetic resonance image, a 3D U-net was employed to obtain the automatic hippocampus segmentation at each hemisphere. Secondly, ASM was performed on a group of pre-obtained template surfaces to generate mean shape and shape variation parameters through principal component analysis. Ultimately, hybrid particle swarm optimization was utilized to search …
