A new whole-sentence language model - neural trans-dimensional random field language model (neural TRF LM), where sentences are modeled as a collection of random fields, and the potential function is defined by a neural network, has been introduced and successfully trained by noise-contrastive estimation (NCE). In this paper, we extend NCE and propose dynamic noise-contrastive estimation (DNCE) to solve the two problems observed in NCE training. First, a dynamic noise distribution is introduced and trained simultaneously to converge to the data distribution. This helps to significantly cut down the noise sample number used in NCE and reduce the training cost. Second, DNCE discriminates between sentences generated from the noise distribution and sentences generated from the interpolation of the data distribution and the noise distribution. This alleviates the overfitting problem caused by the sparseness of the training set. With DNCE, we can successfully and efficiently train neural TRF LMs on large corpus (about 0.8 billion words) with large vocabulary (about 568 K words). Neural TRF LMs perform as good as LSTM LMs with less parameters and being 5x~114x faster in rescoring sentences. Interpolating neural TRF LMs with LSTM LMs and n-gram LMs can further reduce the error rates.
Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);The University of New South Wales(新南威尔士大学)
Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Information
Dept. of Electrical and Computer Engineering, Seoul National University(首尔大学电气与计算机工程系);Automation and Systems Research Institute, Seoul National University(首尔大学自动化与系统研究所)
Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer
Stevens Institute of Technology(史蒂文斯理工学院);San Francisco State University(旧金山州立大学);University of Minnesota(明尼苏达大学);The Chinese University of Hong Kong(香港中文大学);New York University(纽约大学)