PRHLT Research Center, Universitat Politècnica de València（PRHLT研究中心，瓦伦西亚理工大学）；AUSY R&D（奥西研发中心）；IRIT, CNRS, Université de Toulouse（图卢兹大学计算机研究所，法国国家科学研究中心）
Bilal Ghanem, Jihen Karoui, Farah Benamara, Paolo Rosso, Véronique Moriceau
This paper proposes the first multilingual (French, English and Arabic) and multicultural (Indo-European languages vs. less culturally close languages) irony detection system. We employ both feature-based models and neural architectures using monolingual word representation. We compare the performance of these systems with state-of-the-art systems to identify their capabilities. We show that these monolingual models trained separately on different languages using multilingual word representation or text-based features can open the door to irony detection in languages that lack of annotated data for irony.
Multilingual Irony Detection with Dependency Syntax and Neural Models
Università degli Studi di Torino（都灵大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；Università degli Studi di Cagliari（卡利亚里大学）；Université de Toulouse（图卢兹大学）
Baidu（百度）；Shanghai Pudong Development Bank（上海浦东发展银行）
Detecting sarcasm and verbal irony from people's subjective statements is crucial to understanding their intended meanings and real sentiments and positions in social scenarios. This paper describes the X-PuDu system that participated in SemEval-2022 Task 6, iSarcasmEval - Intended Sarcasm Detection in English and Arabic, which aims at detecting intended sarcasm in various settings of natural language understanding. Our solution finetunes pre-trained language models, such as ERNIE-M and DeBERTa, under the multilingual settings to recognize the irony from Arabic and English texts. Our system ra…
University of Innsbruck（因斯布鲁克大学）；The University of Tokyo（东京大学）
Predicting context-dependent and non-literal utterances like sarcastic and ironic expressions still remains a challenging task in NLP, as it goes beyond linguistic patterns, encompassing common sense and shared knowledge as crucial components. To capture complex morpho-syntactic features that can usually serve as indicators for irony or sarcasm across dynamic contexts, we propose a model that uses character-level vector representations of words, based on ELMo. We test our model on 7 different datasets derived from 3 different data sources, providing state-of-the-art performance in 6 of them, a…
Supervised deep learning requires large amounts of training data. In the context of the FIRE2019 Arabic irony detection shared task (IDAT@FIRE2019), we show how we mitigate this need by fine-tuning the pre-trained bidirectional encoders from transformers (BERT) on gold data in a multi-task setting. We further improve our models by by further pre-training BERT on `in-domain' data, thus alleviating an issue of dialect mismatch in the Google-released BERT model. Our best model acquires 82.4 macro F1 score, and has the unique advantage of being feature-engineering free (i.e., based exclusively on …
University of Pennsylvania（宾夕法尼亚大学）；IBM Research（IBM研究院）
We develop high performance multilingualAbstract Meaning Representation (AMR) sys-tems by projecting English AMR annotationsto other languages with weak supervision. Weachieve this goal by bootstrapping transformer-based multilingual word embeddings, in partic-ular those from cross-lingual RoBERTa (XLM-R large). We develop a novel technique forforeign-text-to-English AMR alignment, usingthe contextual word alignment between En-glish and foreign language tokens. This wordalignment is weakly supervised and relies onthe contextualized XLM-R word embeddings.We achieve a highly competitive performa…
Sarcasm is a form of irony that involves saying or writing something that is opposite or opposite to what one really means, often in a humorous or mocking way. It is often used to mock or mock someone or something, or to be humorous or amusing. Sarcasm is usually conveyed through tone of voice, facial expressions, or other forms of nonverbal communication, but it can also be indicated by the use of certain words or phrases that are typically associated with irony or humor. Sarcasm detection is difficult because it relies on context and non-verbal cues. It can also be culturally specific, subje…
Sarcasm detection is the task of identifying irony containing utterances in sentiment-bearing text. However, the figurative and creative nature of sarcasm poses a great challenge for affective computing systems performing sentiment analysis. This article compiles and reviews the salient work in the literature of automatic sarcasm detection. Thus far, three main paradigm shifts have occurred in the way researchers have approached this task: 1) semi-supervised pattern extraction to identify implicit sentiment, 2) use of hashtag-based supervision, and 3) incorporation of context beyond target tex…
The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis
Saarland University（萨尔大学）；Bosch Center for AI（博世人工智能中心）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Eastern Institute of Technology（东方理工学院）；Mila（米拉研究所）；McGill University（麦吉尔大学）
When Benchmarks Mislead: Shortcut Learning, Length Confounds, and the Limits of Cross-Dataset Generalization in Multilingual Fake News and Sarcasm Detection
LuxVeri at GenAI Detection Task 1: Inverse Perplexity Weighted Ensemble for Robust Detection of AI-Generated Text across English and Multilingual Contexts
Shahjalal University of Science and Technology（沙赫贾拉勒科技大学）；University of Alberta（阿尔伯塔大学）
On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generation
Center for Language and Cognition (CLCG), University of Groningen（格罗宁根大学语言与认知中心）；Institute for Logic, Language and Computation (ILLC), University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）
CSWBench: Evaluating Context-Sensitive Code-Switching in Creative Writing
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；College of Software, Jilin University（吉林大学软件学院）；Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.（中国电信人工智能技术（北京）有限公司星辰AGI实验室）；Bilibili Inc.（哔哩哔哩公司）；University of Ottawa（渥太华大学）；International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
RIKEN AIP（理化学研究所先进智能项目中心）；Tohoku University（东北大学）；Heidelberg Institute for Theoretical Studies gGmbH（海德堡理论研究所）
Pretrained contextual and non-contextual subword embeddings have become available in over 250 languages, allowing massively multilingual NLP. However, while there is no dearth of pretrained embeddings, the distinct lack of systematic evaluations makes it difficult for practitioners to choose between them. In this work, we conduct an extensive evaluation comparing non-contextual subword embeddings, namely FastText and BPEmb, and a contextual representation method, namely BERT, on multilingual named entity recognition and part-of-speech tagging. We find that overall, a combination of BERT, BPEmb…
The Hong Kong University of Science and Technology（香港科技大学）；Google Brain（谷歌大脑）；ML Collective（机器学习集体）
General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (NLP) tasks and benchmarks when inferring instructions from very few examples. Here, we evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages without any parameter updates. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones. Finally, we find the in-co…
Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, l…
We describe the Uppsala NLP submission to SemEval-2021 Task 2 on multilingual and cross-lingual word-in-context disambiguation. We explore the usefulness of three pre-trained multilingual language models, XLM-RoBERTa (XLMR), Multilingual BERT (mBERT) and multilingual distilled BERT (mDistilBERT). We compare these three models in two setups, fine-tuning and as feature extractors. In the second case we also experiment with using dependency-based information. We find that fine-tuning is better than feature extraction. XLMR performs better than mBERT in the cross-lingual setting both with fine-tun…
Multilingual Idioms in Sentences and Conversations Across High-, Medium-, and Low-Resource Languages
Mohamed bin Zayed University of Artificial Intelligence（莫扎德大学人工智能大学）；University of Hamburg（汉堡大学）；Manipal University（曼印大学）；IIIT Hyderabad（海得拉尔IIIT）；University of Science and Technology of Hanoi（河内科学技术大学）；Universitas Islam Indonesia（印尼伊斯兰大学）；Princeton University（普林斯顿大学）
This paper documents the Team Copenhagen system which placed first in the CoNLL--SIGMORPHON 2018 shared task on universal morphological reinflection, Task 2 with an overall accuracy of 49.87. Task 2 focuses on morphological inflection in context: generating an inflected word form, given the lemma of the word and the context it occurs in. Previous SIGMORPHON shared tasks have focused on context-agnostic inflection---the "inflection in context" task was introduced this year. We approach this with an encoder-decoder architecture over character sequences with three core innovations, all contributi…
The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-ofthe-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multip…
College of Electrical Engineering and Computer Science, National Taiwan University（台湾大学电机资讯学院）
Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability. We also observe the language-specific information in multilingual BERT. By manipulating the latent representations, we can control t…
Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心（CIS））
Word embeddings induced from local context are prevalent in NLP. A simple and effective context-based multilingual embedding learner is Levy et al. (2017)'s S-ID (sentence ID) method. Another line of work induces high-performing multilingual embeddings from concepts (Dufter et al., 2018). In this paper, we propose Co+Co, a simple and scalable method that combines context-based and concept-based learning. From a sentence aligned corpus, concepts are extracted via sampling; words are then associated with their concept ID and sentence ID in embedding learning. This is the first work that successf…
Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models
Research Center Trustworthy Data Science and Security of the University Alliance Ruhr（鲁尔大学联盟可信数据科学与安全研究中心）；Faculty of Informatics, Technical University Dortmund（多特蒙德工业大学信息学院）
Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
Institute for Analytics and Data Science, University of Essex（数据分析与科学研究所，埃塞克斯大学）；School of Computer Science and Electronic Engineering, University of Essex（计算机科学与电子工程学院，埃塞克斯大学）
Amazonas State Univ.（亚马孙州立大学）；Federal Univ. of Amazonas（亚马孙联邦大学）
This paper describes the system submitted by our team (BabelEnconding) to SemEval-2020 Task 3: Predicting the Graded Effect of Context in Word Similarity. We propose an approach that relies on translation and multilingual language models in order to compute the contextual similarity between pairs of words. Our hypothesis is that evidence from additional languages can leverage the correlation with the human generated scores. BabelEnconding was applied to both subtasks and ranked among the top-3 in six out of eight task/language combinations and was the highest scoring system three times.
Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Birmingham City University（伯明翰城市大学）；University of Wolverhampton（伍尔弗汉普顿大学）
Identifying whether a word carries the same meaning or different meaning in two contexts is an important research area in natural language processing which plays a significant role in many applications such as question answering, document summarisation, information retrieval and information extraction. Most of the previous work in this area rely on language-specific resources making it difficult to generalise across languages. Considering this limitation, our approach to SemEval-2021 Task 2 is based only on pretrained transformer models and does not use any language-specific processing and res…
We present a neural network architecture based on bidirectional LSTMs to compute representations of words in the sentential contexts. These context-sensitive word representations are suitable for, e.g., distinguishing different word senses and other context-modulated variations in meaning. To learn the parameters of our model, we use cross-lingual supervision, hypothesizing that a good representation of a word in context will be one that is sufficient for selecting the correct translation into a second language. We evaluate the quality of our representations as features in three downstream tas…
Previous work on cross-lingual sequence labeling tasks either requires parallel data or bridges the two languages through word-byword matching. Such requirements and assumptions are infeasible for most languages, especially for languages with large linguistic distances, e.g., English and Chinese. In this work, we propose a Multilingual Language Model with deep semantic Alignment (MLMA) to generate language-independent representations for cross-lingual sequence labeling. Our methods require only monolingual corpora with no bilingual resources at all and take advantage of deep contextualized rep…
University of California, Berkeley（加州大学伯克利分校）；Computer Science Division（计算机科学系）
We propose procedures for evaluating and strengthening contextual embedding alignment and show that they are useful in analyzing and improving multilingual BERT. In particular, after our proposed alignment procedure, BERT exhibits significantly improved zero-shot performance on XNLI compared to the base model, remarkably matching pseudo-fully-supervised translate-train models for Bulgarian and Greek. Further, to measure the degree of alignment, we introduce a contextual version of word retrieval and show that it correlates well with downstream zero-shot transfer. Using this word retrieval task…
