arXiv 1704.045652017-04cs.CL

Neural Paraphrase Identification of Questions with Noisy Pretraining

Google Inc.（谷歌公司）

Gaurav Singh Tomar, Thyago Duque, Oscar Täckström, Jakob Uszkoreit, Dipanjan Das

英文摘要：

We present a solution to the problem of paraphrase identification of questions. We focus on a recent dataset of question pairs annotated with binary paraphrase labels and show that a variant of the decomposable attention model (Parikh et al., 2016) results in accurate performance on this task, while being far simpler than many competing neural architectures. Furthermore, when the model is pretrained on a noisy dataset of automatically collected question paraphrases, it obtains the best reported performance on the dataset.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Generative Pretraining for Paraphrase Evaluation

Novoic

We introduce ParaBLEU, a paraphrase representation learning model and evaluation metric for text generation. Unlike previous approaches, ParaBLEU learns to understand paraphrasis using generative conditioning as a pretraining objective. ParaBLEU correlates more strongly with human judgements than existing metrics, obtaining new state-of-the-art results on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only $50\%$ of the available training data and surpassing BLEU, ROUGE and METEOR with only $40$ labelle…

2021-07-24cs.CLcs.LG

Paraphrase Detection on Noisy Subtitles in Six Languages

University of Helsinki（赫尔辛基大学）

We perform automatic paraphrase detection on subtitle data from the Opusparcus corpus comprising six European languages: German, English, Finnish, French, Russian, and Swedish. We train two types of supervised sentence embedding models: a word-averaging (WA) model and a gated recurrent averaging network (GRAN) model. We find out that GRAN outperforms WA and is more robust to noisy training data. Better results are obtained with more and noisier data than less and cleaner data. Additionally, we experiment on other datasets, without reaching the same level of performance, because of domain misma…

2018-09-21cs.CL

Large Scale Question Paraphrase Retrieval with Smoothed Deep Metric Learning

University of Trento（特伦托大学）；Amazon Alexa（亚马逊Alexa）

The goal of a Question Paraphrase Retrieval (QPR) system is to retrieve equivalent questions that result in the same answer as the original question. Such a system can be used to understand and answer rare and noisy reformulations of common questions by mapping them to a set of canonical forms. This has large-scale applications for community Question Answering (cQA) and open-domain spoken language question answering systems. In this paper we describe a new QPR system implemented as a Neural Information Retrieval (NIR) system consisting of a neural network sentence encoder and an approximate k-…

2019-05-29cs.AIcs.CLcs.IR

A Data-Driven Investigation of Noise-Adaptive Utterance Generation with Linguistic Modification

Saarland University（萨尔大学）

In noisy environments, speech can be hard to understand for humans. Spoken dialog systems can help to enhance the intelligibility of their output, either by modifying the speech synthesis (e.g., imitate Lombard speech) or by optimizing the language generation. We here focus on the second type of approach, by which an intended message is realized with words that are more intelligible in a specific noisy environment. By conducting a speech perception experiment, we created a dataset of 900 paraphrases in babble noise, perceived by native English speakers with normal hearing. We find that careful…

2022-10-19cs.CLcs.SDeess.AS

TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data

Alibaba Group（阿里巴巴集团）

Among ubiquitous multimodal data in the real world, text is the modality generated by human, while image reflects the physical world honestly. In a visual understanding application, machines are expected to understand images like human. Inspired by this, we propose a novel self-supervised learning method, named Text-enhanced Visual Deep InfoMax (TVDIM), to learn better visual representations by fully utilizing the naturally-existing multimodal data. Our core idea of self-supervised learning is to maximize the mutual information between features extracted from multiple views of a shared context…

2021-06-13cs.CL

Robust Named Entity Recognition with Truecasing Pretraining

University of Pennsylvania（宾夕法尼亚大学）

Although modern named entity recognition (NER) systems show impressive performance on standard datasets, they perform poorly when presented with noisy data. In particular, capitalization is a strong signal for entities in many languages, and even state of the art models overfit to this feature, with drastically lower performance on uncapitalized text. In this work, we address the problem of robustness of NER systems in data with noisy or uncertain casing, using a pretraining objective that predicts casing in text, or a truecaser, leveraging unlabeled data. The pretrained truecaser is combined …

2019-12-15cs.CL

更多相关论文收起更多论文

Pretraining the Noisy Channel Model for Task-Oriented Dialogue

University of Oxford（牛津大学）；DeepMind

Direct decoding for task-oriented dialogue is known to suffer from the explaining-away effect, manifested in models that prefer short and generic responses. Here we argue for the use of Bayes' theorem to factorize the dialogue task into two models, the distribution of the context given the response, and the prior for the response itself. This approach, an instantiation of the noisy channel model, both mitigates the explaining-away effect and allows the principled incorporation of large pretrained models for the response prior. We present extensive experiments showing that a noisy channel model…

2021-03-18cs.AIcs.CLcs.LG

Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining

University of Edinburgh（爱丁堡大学）；WS AI Labs（WS人工智能实验室）；New York University（纽约大学）

Advances in neural modeling have achieved state-of-the-art (SOTA) results on public natural language processing (NLP) benchmarks, at times surpassing human performance. However, there is a gap between public benchmarks and real-world applications where noise, such as typographical or grammatical mistakes, is abundant and can result in degraded performance. Unfortunately, works which evaluate the robustness of neural models on noisy data and propose improvements, are limited to the English language. Upon analyzing noise in different languages, we observe that noise types vary greatly across lan…

2023-02-10cs.AIcs.CLcs.LG

WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web

Google Brain（谷歌大脑）；Google（谷歌公司）

Relation extraction is used to populate knowledge bases that are important to many applications. Prior datasets used to train relation extraction models either suffer from noisy labels due to distant supervision, are limited to certain domains or are too small to train high-capacity models. This constrains downstream applications of relation extraction. We therefore introduce: WebRED (Web Relation Extraction Dataset), a strongly-supervised human annotated dataset for extracting relationships from a variety of text found on the World Wide Web, consisting of ~110K examples. We also describe the …

2021-02-18cs.CLcs.IR

当少即是多：大规模探究预训练 LLM 的数据剪枝

When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale

Cohere for AI（Cohere人工智能研究院）；Cohere（Cohere公司）

本研究探索了大规模预训练LLM的数据剪枝方法，系统比较了困惑度、Error L2-Norm和记忆化等数据质量评估指标，发现简单的困惑度方法表现最优，仅用30%原始数据即可超越无剪枝基线。

2023-09-08cs.CLcs.LG

通过过滤和选择改进大规模弱监督ASR

Improving Large-Scale Weakly Supervised ASR by Filtering and Selection

NTT Corporation（日本NTT公司）；Kyoto University（京都大学）；Graduate School of Informatics（信息学研究生院）

针对弱监督ASR中的噪声标签和领域不匹配问题，提出基于字符错误率过滤和声学相似性选择的训练方法，在9万小时日语数据上实现6.4%和4.0%的CER降低。

2026-06-27cs.CLeess.AS

Data Weighted Training Strategies for Grammatical Error Correction

Google Research（谷歌研究院）

Recent progress in the task of Grammatical Error Correction (GEC) has been driven by addressing data sparsity, both through new methods for generating large and noisy pretraining data and through the publication of small and higher-quality finetuning data in the BEA-2019 shared task. Building upon recent work in Neural Machine Translation (NMT), we make use of both kinds of data by deriving example-level scores on our large pretraining data based on a smaller, higher-quality dataset. In this work, we perform an empirical study to discover how to best incorporate delta-log-perplexity, a type of…

2020-09-09cs.CLstat.ML

Nemotron-CC-Math：一个1330亿token规模的高质量数学预训练数据集

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset

NVIDIA（英伟达）；Boston University（波士顿大学）

本文提出Nemotron-CC-Math，通过布局感知渲染与LLM清洗的流水线从Common Crawl中鲁棒提取数学内容，构建了133B token的高质量数学预训练语料库，在数学与代码推理任务上显著超越现有开放数据集。

2025-08-20cs.AIcs.CLcs.LG

Variational Reasoning for Question Answering with Knowledge Graph

College of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）；Amazon Web Services（亚马逊云科技）

Knowledge graph (KG) is known to be helpful for the task of question answering (QA), since it provides well-structured relational information between entities, and allows one to further infer indirect facts. However, it is challenging to build QA systems which can learn to reason over knowledge graphs based on question-answer pairs alone. First, when people ask questions, their expressions are noisy (for example, typos in texts, or variations in pronunciations), which is non-trivial for the QA system to match those mentioned entities to the knowledge graph. Second, many questions require multi…

2017-11-27cs.AIcs.CLcs.LG