The problem of rare and unknown words is an important issue that can potentially influence the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context.~We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known.~We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.
A New NMT Model for Translating Clinical Texts from English to Spanish
College of Information and Computer Science, University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特信息与计算机科学学院);Department of Computer Sciences, University of Massachusetts Lowell(马萨诸塞大学洛约拉计算机科学系);Department of Quantitative Health Sciences, University of Massachusetts Medical School(马萨诸塞大学医学学院定量健康科学系);Bedford VAMC and CHOIR(贝德福德退伍军人医疗中心和CHOIR)
How much does a word weigh? Weighting word embeddings for word sense induction
Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);Samsung Moscow Research Center(三星莫斯科研究中心);University of Hamburg(汉堡大学)
本文描述了我们参与第一个关于俄语RUSSE'2018词义归纳和消歧的共享任务(Panchenko等人,2018年)。对于几十个模棱两可的单词中的每一个,参与者被要求根据这个单词的含义对包含它的文本片段进行分组,这是事先没有提供的,因此是任务的“归纳”部分。例如,给出了一个单词“bank”和一组出现该单词的文本片段(也称为“context”),例如“bank is a financial institution that accepts deposit”和“river bank is a slope beside a body of water”。一名参与者被要求在未知的预先数量的集群对应的集群,在这种情况下,“公司”和“地区”的意义的单词“银行”这样的情况下进行聚类。组织者提出了三个不同复杂性和文本类型的评估数据集,分别基于维基百科,网页和俄语词典的文本。我们提出了两个实验:分别基于表示为词嵌入的加权平均的上下文的聚类和基于使用两个最先进的产生式神经机器翻译系统的机器翻译的正和负两个。我们的团队在18个参与团队中的两个数据集上显示了第二好的结果,在剩下的一个数据集上显示了第三好的结果。我们成功地大幅超越了前几年基于感觉嵌入的竞争性最先进的基线。
A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
Preferred Networks, Inc.(Preferred Networks 公司);Tokyo Institute of Technology(东京工业大学);Tohoku University(东北大学);RIKEN(理化学研究所)
本研究针对神经语言模型中使用的词嵌入方法来识别语篇中未知词或实体的意义。我们提出了一种通过跟踪上下文在神经模型的输入和输出层中动态构建和利用词嵌入的方法。这扩展了Kobayashi et al.(2016)中使用的动态实体表示,并结合了Gu et al.(2016)和Gulcehre et al.(2016)独立提出的复制机制。此外,我们还构建了一个新的任务和数据集,称为Ancesturized Language Modeling,用于评估阅读时捕捉词义的能力。使用我们的新数据集进行的实验表明,所提出的RNN语言模型变体优于基线模型。此外,实验还表明,输出层的动态更新有助于模型预测再现实体,而输入层的动态更新则可有效预测再现实体后面的单词。
FullStop:Punctuation and Segmentation Prediction for Dutch with Transformers
Instituut voor de Nederlandse Taal(荷兰语言研究所);KU Leuven(鲁汶大学);Hochschule für Technik und Wirtschaft(德累斯顿工程与经济应用科学大学)
当对比利时荷兰语应用自动语音识别(ASR)时(Van Dyck et al. 2021),输出由未分段的单词流组成,没有任何标点符号。下一步是执行分段和插入标点符号,使ASR输出更具可读性,易于手动更正。据我们所知,没有一个公共可用的标点符号插入系统的荷兰语,功能在一个可用的水平。我们在这里提出的模型是Guhr等人(2021)荷兰模型的扩展,并公开提供。我们基于荷兰语言模型RobBERT(Delobelle et al. 2020)训练了一个序列分类模型。对于输入序列中的每个单词,模型都会预测该单词后面的标点符号。我们还扩展了多语言模型,用于语言未知或代码切换适用的情况。当执行分割任务时,将最佳模型应用到域外测试数据上,将ASR输出流的200个单词的滑动窗口发送到分类器,并且当系统预测具有高于阈值的比率的分割标点符号时应用分割。结果表明,它比机器翻译基线方法要好得多。
Hierarchical Memory Networks for Answer Selection on Unknown Words
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Center for Excellence in Brain Science and Intelligence Technology, CAS(中国科学院脑科学与智能技术卓越创新中心)
Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models
Tsinghua University(清华大学);University of Washington(华盛顿大学);Groundlight AI;Rice University(莱斯大学);University of Chinese Academy of Sciences(中国科学院大学);Qinghai University(青海大学)
Part of speech and gramset tagging algorithms for unknown words based on morphological dictionaries of the Veps and Karelian languages
Institute of Applied Mathematical Research of the Karelian Research Centre of the Russian Academy of Sciences(俄罗斯科学院卡累利阿研究中心应用数学研究所);Petrozavodsk State University(彼得罗扎沃茨克国立大学);Institute of Linguistics, Literature and History of the Karelian Research Centre of the Russian Academy of Sciences(俄罗斯科学院卡累利阿研究中心语言、文学与历史研究所)
Probing the statistical properties of unknown texts: application to the Voynich Manuscript
Institute of Physics of São Carlos, University of São Paulo(圣保罗大学圣保罗卡洛斯物理研究所);Max Planck Institute for the Physics of Complex Systems (MPIPKS)(马克斯·普朗克复杂系统物理研究所);Potsdam Institute for Climate Impact Research (PIK)(波茨坦气候影响研究所)
Joint Chinese Word Segmentation and Part-of-speech Tagging via Two-stage Span Labeling
Multimedia Communications Laboratory, University of Information Technology(信息技术大学多媒体通信实验室);University of Information Technology(信息技术大学);Vietnam National University(越南国立大学)