News articles, image captions, product reviews and many other texts mention people and organizations whose name recognition could vary for different audiences. In such cases, background information about the named entities could be provided in the form of an appositive noun phrase, either written by a human or generated automatically. We expand on the previous work in appositive generation with a new, more realistic, end-to-end definition of the task, instantiated by a dataset that spans four languages (English, Spanish, German and Polish), two entity types (person and organization) and two domains (Wikipedia and News). We carry out an extensive analysis of the data and the task, pointing to the various modeling challenges it poses. The results we obtain with standard language generation methods show that the task is indeed non-trivial, and leaves plenty of room for improvement.
In this work, we introduce X-FACT: the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers. The dataset includes a multilingual evaluation benchmark that measures both out-of-domain generalization, and zero-shot capabilities of the multilingual models. Using state-of-the-art multilingual transformer-based models, we develop several automated fact-checking models that, along with textual claims, make use of additional metadata an…
Evaluating Multilingual Sentence Embeddings for Translation Error Detection:An English--Greek Contrastive Study
Ionian University（爱奥尼亚大学）；Laboratory on Digital Libraries and Electronic Publishing（数字图书馆与电子出版实验室）；Centre for Literary Translation and Book Policy（文学翻译与图书政策中心）
Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as "Punta Cana is located in _." However, while knowledge is both written and queried in many languages, studies on LMs' factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for 23 typologically diverse languages. To properly handle language variations, we expand probing methods from single- to mu…
Tracing Multilingual Factual Knowledge Acquisition in Pretraining
Center for Information and Language Processing, LMU Munich（信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Bosch Center for Artificial Intelligence（博世人工智能中心）；Sorbonne Université, CNRS, ISIR, France（索邦大学，法国国家科学研究中心，ISIR）
Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
Bosch Center for Artificial Intelligence（博世人工智能中心）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Hochschule der Medien（斯图加特媒体大学）；Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用技术大学）
Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge
Indian Institute of Technology Delhi（印度理工学院德里）；University of Washington（华盛顿大学）；Lila Sciences（Lila科学）；University of California, Santa Barbara（加州大学圣芭芭拉分校）
Information Sciences Institute, University of Southern California（南加州大学信息科学研究所）；University of California, Los Angeles（加州大学洛杉矶分校）
News article revision histories provide clues to narrative and factual evolution in news articles. To facilitate analysis of this evolution, we present the first publicly available dataset of news revision histories, NewsEdits. Our dataset is large-scale and multilingual; it contains 1.2 million articles with 4.6 million versions from over 22 English- and French-language newspaper sources based in three countries, spanning 15 years of coverage (2006-2021). We define article-level edit actions: Addition, Deletion, Edit and Refactor, and develop a high-accuracy extraction algorithm to identify t…
This study presents a new dataset on rumor detection in Finnish language news headlines. We have evaluated two different LSTM based models and two different BERT models, and have found very significant differences in the results. A fine-tuned FinBERT reaches the best overall accuracy of 94.3% and rumor label accuracy of 96.0% of the time. However, a model fine-tuned on Multilingual BERT reaches the best factual label accuracy of 97.2%. Our results suggest that the performance difference is due to a difference in the original training data. Furthermore, we find that a regular LSTM model works b…
Factual Inconsistencies in Multilingual Wikipedia Tables
CNR ISTC（意大利国家研究委员会信息与通信技术研究所）；Fraunhofer Institute for Applied Information Technology FIT（弗劳恩霍夫应用信息技术研究所）；Tallinn University of Technology（塔林理工大学）；EURECOM（欧瑞康）；Technical University of Munich（慕尼黑技术大学）；University of Amsterdam（阿姆斯特丹大学）
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
Center for Language and Cognition, University of Groningen（格罗宁根大学语言与认知中心）；Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）
Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
Massachusetts Institute of Technology（麻省理工学院）；Havard Univeristy（哈佛大学）；LG CNS；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）；Google（谷歌）；Stanford University（斯坦福大学）
Knowledge-enhanced language representation learning has shown promising results across various knowledge-intensive NLP tasks. However, prior methods are limited in efficient utilization of multilingual knowledge graph (KG) data for language model (LM) pretraining. They often train LMs with KGs in indirect ways, relying on extra entity/relation embeddings to facilitate knowledge injection. In this work, we explore methods to make better use of the multilingual annotation and language agnostic property of KG triples, and present novel knowledge based multilingual language models (KMLMs) trained …
Output Language Confusion under Multilingual Prompt Contamination
University of Tübingen（蒂宾根大学）；Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；Thapar Institute of Engineering & Technology（塔帕尔工程技术学院）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Indian AI Research Organization（印度人工智能研究组织）
Studio Ousia（欧西亚工作室）；The University of Tokyo（东京大学）；RIKEN AIP（理化学研究所人工智能研究中心）
Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages with entity representations and show the model consistently outperforms word-based pretrained models in various cross-lingual transf…
One Mind, Many Tongues: A Deep Dive into Language-Agnostic Knowledge Neurons in Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
Information and Computer Science Department, King Fahd University of Petroleum and Minerals（信息与计算机科学系，国王法赫德石油和矿物大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
On the Entity-Level Alignment in Crosslingual Consistency
Center for Information and Language Processing, LMU Munich（信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））；Sorbonne Université, CNRS, ISIR, France（索邦大学，CNRS，ISIR，法国）
