arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1808.04217cs.CLcs.AI
收藏

基于序列一致性的句子表示无监督学习

Unsupervised Learning of Sentence Representations Using Sequence Consistency

Siddhartha Brahma

摘要:

计算句子的通用分布表示是自然语言处理的一项基本任务。我们提出了ConsSent,一个简单但令人惊讶的强大的无监督方法来学习这样的表示,通过执行一致性约束的令牌序列。我们考虑两类这样的约束-序列,形成一个句子和两个序列之间,形成一个句子合并时。我们通过训练句子编码器来区分一致和不一致的例子,后者是通过六种不同的方式随机扰动一致的例子来生成的。对几个迁移学习和语言探测任务的广泛评估显示,在强无监督和有监督的基线上,性能有所提高,在几种情况下大大超过了它们。我们最好的结果是通过在多任务设置中训练句子编码器和在单个任务上训练的编码器集合来实现的。

英文摘要:

Computing universal distributed representations of sentences is a fundamental task in natural language processing. We propose ConsSent, a simple yet surprisingly powerful unsupervised method to learn such representations by enforcing consistency constraints on sequences of tokens. We consider two classes of such constraints -- sequences that form a sentence and between two sequences that form a sentence when merged. We learn sentence encoders by training them to distinguish between consistent and inconsistent examples, the latter being generated by randomly perturbing consistent examples in six different ways. Extensive evaluation on several transfer learning and linguistic probing tasks shows improved performance over strong unsupervised and supervised baselines, substantially surpassing them in several cases. Our best results are achieved by training sentence encoders in a multitask setting and by an ensemble of encoders trained on the individual tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑