The advent of context-aware NMT has resulted in promising improvements in the overall translation quality and specifically in the translation of discourse phenomena such as pronouns. Previous works have mainly focused on the use of past sentences as context with a focus on anaphora translation. In this work, we investigate the effect of future sentences as context by comparing the performance of a contextual NMT model trained with the future context to the one trained with the past context. Our experiments and evaluation, using generic and pronoun-focused automatic metrics, show that the use of future context not only achieves significant improvements over the context-agnostic Transformer, but also demonstrates comparable and in some cases improved performance over its counterpart trained on past context. We also perform an evaluation on a targeted cataphora test suite and report significant gains over the context-agnostic Transformer in terms of BLEU.
Nanyang Technological University（南洋理工大学）；Salesforce Research Asia（Salesforce亚洲研究院）
Popular Neural Machine Translation model training uses strategies like backtranslation to improve BLEU scores, requiring large amounts of additional data and training. We introduce a class of conditional generative-discriminative hybrid losses that we use to fine-tune a trained machine translation model. Through a combination of targeted fine-tuning objectives and intuitive re-use of the training data the model has failed to adequately learn from, we improve the model performance of both a sentence-level and a contextual model without using any additional data. We target the improvement of pro…
Possessive pronouns are used as determiners in English when no equivalent would be used in a Japanese sentence with the same meaning. This paper proposes a heuristic method of generating such possessive pronouns even when there is no equivalent in the Japanese. The method uses information about the use of possessive pronouns in English treated as a lexical property of nouns, in addition to contextual information about noun phrase referentiality and the subject and main verb of the sentence that the noun phrase appears in. The proposed method has been implemented in NTT Communication Science La…
The Hong Kong University of Science and Technology（香港科技大学）；Tencent AI Lab（腾讯人工智能实验室）
Linking pronominal expressions to the correct references requires, in many cases, better analysis of the contextual information and external knowledge. In this paper, we propose a two-layer model for pronoun coreference resolution that leverages both context and external knowledge, where a knowledge attention mechanism is designed to ensure the model leveraging the appropriate source of external knowledge based on different context. Experimental results demonstrate the validity and effectiveness of our model, where it outperforms state-of-the-art models by a large margin.
Carnegie Mellon University（卡内基梅隆大学）；Instituto Superior Técnico（高等技术学院）；Instituto de Telecomunicações（电信研究所）；Unbabel
Context-aware machine translation models are designed to leverage contextual information, but often fail to do so. As a result, they inaccurately disambiguate pronouns and polysemous words that require context for resolution. In this paper, we ask several questions: What contexts do human translators use to resolve ambiguous words? Are models paying large amounts of attention to the same context? What if we explicitly train them to do so? To answer these questions, we introduce SCAT (Supporting Context for Ambiguous Translations), a new English-French dataset comprising supporting context word…
Failure of contextual invariance in large language models
Network Science Institute, Northeastern University（网络科学研究所，东北大学）；Center for Health Informatics Program, Boston Children’s Hospital（健康信息学计划中心，波士顿儿童医院）；Dept. of Mathematics, City St George’s, University of London（伦敦大学城市圣乔治学院数学系）；IT University of Copenhagen（哥本哈根IT大学）
Research Center for Social Computing and Information Retrieval（社会计算与信息检索研究中心）；Harbin Institute of Technology（哈尔滨工业大学）
Existing approaches for Chinese zero pronoun resolution overlook semantic information. This is because zero pronouns have no descriptive information, which results in difficulty in explicitly capturing their semantic similarities with antecedents. Moreover, when dealing with candidate antecedents, traditional systems simply take advantage of the local information of a single candidate antecedent while failing to consider the underlying information provided by the other candidates from a global perspective. To address these weaknesses, we propose a novel zero pronoun-specific neural network, wh…
The pre-trained BERT model achieves a remarkable state of the art across a wide range of tasks in natural language processing. For solving the gender bias in gendered pronoun resolution task, I propose a novel neural network model based on the pre-trained BERT. This model is a type of mention score classifier and uses an attention mechanism with no parameters to compute the contextual representation of entity span, and a vector to represent the triple-wise semantic similarity among the pronoun and the entities. In stage 1 of the gendered pronoun resolution task, a variant of this model, traine…
Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling
MAIS, Institute of Automation, Chinese Academy of Sciences Beijing China；School of AI, University of\ Academy of Sciences Beijing China；Beijing Jiaotong University Beijing China；Geely AI lab Ningbo Zhejiang China；MAIS, Institute of Automation, Chinese Academy of Sciences；School of AI, University of\ Academy of Sciences；Beijing Jiaotong University；Geely AI lab
Contextual morphologically-guided tokenization for Latin encoder models
Manning College of Information & Computer Sciences, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校信息与计算机科学学院）；Institute for the Study of the Ancient World, New York University（纽约大学古代世界研究所）
This paper proposes two intuitive metrics, skew and stereotype, that quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. We find evidence that gender stereotype correlates approximately negatively with gender skew in out-of-the-box models, suggesting that there is a trade-off between these two forms of bias. We investigate two methods to mitigate bias. The first approach is an online method which is effective at removing skew at the expense of stereotype. The second, inspired by previous work on ELMo, involves the fine-…
