视觉问答（Visual Question Answering, VQA）是自然语言处理（natural language processing, NLP）与计算机视觉（computer vision, CV）领域的一项挑战性任务，受到研究者的广泛关注。英语作为资源丰富的语言，其视觉问答领域的数据集与模型已取得诸多进展，其他语言的视觉问答也需在资源和模型层面得到发展。此外，目前尚无针对特定国家本土事物与文化特征视觉内容的多语言数据集。为弥补这一不足，我们向研究界提供名为EVJVQA的基准数据集，该数据集包含越南语、英语、日语三种语言的33000余条问答对，基于约5000张拍摄于越南的图像构建，用于评估多语言视觉问答系统或模型。EVJVQA被用作第9届越南语言与语音处理研讨会（VLSP 2022）多语言视觉问答挑战赛的基准数据集，该任务吸引了来自多所高校和机构的62支参赛队伍。本文介绍了挑战赛的组织细节、参赛队伍所采用方法的概述及比赛结果。在私有测试集上，最高性能为F1值0.4392、BLUE值0.4009。前两名队伍提出的多语言问答系统均采用ViT作为预训练视觉模型、mT5作为预训练语言模型，后者是基于Transformer架构的强大预训练语言模型。EVJVQA是一项具有挑战性的数据集，能够推动NLP与CV领域研究者进一步探索面向视觉问答系统的多语言模型或系统。我们已在Codalab评估系统上发布该挑战赛，以供后续研究使用。
英文摘要：
Visual Question Answering (VQA) is a challenging task of natural language processing (NLP) and computer vision (CV), attracting significant attention from researchers. English is a resource-rich language that has witnessed various developments in datasets and models for visual question answering. Visual question answering in other languages also would be developed for resources and models. In addition, there is no multilingual dataset targeting the visual content of a particular country with its own objects and cultural characteristics. To address the weakness, we provide the research community with a benchmark dataset named EVJVQA, including 33,000+ pairs of question-answer over three languages: Vietnamese, English, and Japanese, on approximately 5,000 images taken from Vietnam for evaluating multilingual VQA systems or models. EVJVQA is used as a benchmark dataset for the challenge of multilingual visual question answering at the 9th Workshop on Vietnamese Language and Speech Processing (VLSP 2022). This task attracted 62 participant teams from various universities and organizations. In this article, we present details of the organization of the challenge, an overview of the methods employed by shared-task participants, and the results. The highest performances are 0.4392 in F1-score and 0.4009 in BLUE on the private test set. The multilingual QA systems proposed by the top 2 teams use ViT for the pre-trained vision model and mT5 for the pre-trained language model, a powerful pre-trained language model based on the transformer architecture. EVJVQA is a challenging dataset that motivates NLP and CV researchers to further explore the multilingual models or systems for visual question answering systems. We released the challenge on the Codalab evaluation system for further research.
Technical University of Darmstadt（达姆施塔特工业大学）；New York University（纽约大学）；University of Cambridge（剑桥大学）
Recent advances in multimodal vision and language modeling have predominantly focused on the English language, mostly due to the lack of multilingual multimodal datasets to steer modeling efforts. In this work, we address this gap and provide xGQA, a new multilingual evaluation benchmark for the visual question answering task. We extend the established English GQA dataset to 7 typologically diverse languages, enabling us to detect and explore crucial challenges in cross-lingual visual question answering. We further propose new adapter-based approaches to adapt multimodal transformer-based mode…
Pre-trained language-vision models have shown remarkable performance on the visual question answering (VQA) task. However, most pre-trained models are trained by only considering monolingual learning, especially the resource-rich language like English. Training such models for multilingual setups demand high computing resources and multilingual language-vision dataset which hinders their application in practice. To alleviate these challenges, we propose a knowledge distillation approach to extend an English language-vision model (teacher) into an equally effective multilingual and code-mixed m…
Pre-trained models with dual and cross encoders have shown remarkable success in propelling the landscape of several tasks in vision and language in Visual Question Answering (VQA). However, since they are limited by the requirements of gold annotated data, most of these advancements do not see the light of day in other languages beyond English. We aim to address this problem by introducing a curriculum based on the source and target language translations to finetune the pre-trained models for the downstream task. Experimental results demonstrate that script plays a vital role in the performan…
Department of Computational Linguistics, University of Zurich（苏黎世大学计算语言学系）
While several benefits were realized for multilingual vision-language pretrained models, recent benchmarks across various tasks and languages showed poor cross-lingual generalisation when multilingually pre-trained vision-language models are applied to non-English data, with a large gap between (supervised) English performance and (zero-shot) cross-lingual transfer. In this work, we explore the poor performance of these models on a zero-shot cross-lingual visual question answering (VQA) task, where models are fine-tuned on English visual-question data and evaluated on 7 typologically diverse l…
Can MLLMs Generalize to Multi-Party dialog? Exploring Multilingual Response Generation in Complex Scenarios
School of Computer Science and Engineering, Northwestern Polytechnical University（计算机科学与工程学院，西北工业大学）；School of Computer Science and Technology, Xidian University（计算机科学与技术学院，西安电子科技大学）；School of Artificial Intelligence and Big Data, Henan University of Technology（人工智能与大数据学院，河南工业大学）
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
Capital One；SEACrowd；NAIST；ITB；UofT；KAIST；JAIST；Cardiff University；Independent；HKUST；SMU；Masakhane；University of Lagos；HK PolyU；Boston University；Columbia University；MBZUAI；Monash University；UCSB；Tokyo Tech；AI Singapore；NICT；Cohere；McGill；MILA；Canada CIFAR AI Chair；Ontario Tech
IBM Research AI（IBM研究院AI）；Thomas J. Watson Research Center（托马斯·J·沃森研究中心）
Recent approaches have exploited weaknesses in monolingual question answering (QA) models by adding adversarial statements to the passage. These attacks caused a reduction in state-of-the-art performance by almost 50%. In this paper, we are the first to explore and successfully attack a multilingual QA (MLQA) system pre-trained on multilingual BERT using several attack strategies for the adversarial statement reducing performance by as much as 85%. We show that the model gives priority to English and the language of the question regardless of the other languages in the QA pair. Further, we als…
Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). Answers are based on a heavily curated, language-independent data representation, making results comparable across languages and independent of language-specific passages. With 26 languages, this dataset supplies the widest range of languages to-date for evalu…
We introduce Mintaka, a complex, natural, and multilingual dataset designed for experimenting with end-to-end question-answering models. Mintaka is composed of 20,000 question-answer pairs collected in English, annotated with Wikidata entities, and translated into Arabic, French, German, Hindi, Italian, Japanese, Portuguese, and Spanish for a total of 180,000 samples. Mintaka includes 8 types of complex questions, including superlative, intersection, and multi-hop questions, which were naturally elicited from crowd workers. We run baselines over Mintaka, the best of which achieves 38% hits@1 i…
Recently, multilingual question answering became a crucial research topic, and it is receiving increased interest in the NLP community. However, the unavailability of large-scale datasets makes it challenging to train multilingual QA systems with performance comparable to the English ones. In this work, we develop the Translate Align Retrieve (TAR) method to automatically translate the Stanford Question Answering Dataset (SQuAD) v1.1 to Spanish. We then used this dataset to train Spanish QA systems by fine-tuning a Multilingual-BERT model. Finally, we evaluated our QA models with the recently …
The goal of the paper is to predict answers to questions given a passage of Qur'an. The answers are always found in the passage, so the task of the model is to predict where an answer starts and where it ends. As the initial data set is rather small for training, we make use of multilingual BERT so that we can augment the training data by using data available for languages other than Arabic. Furthermore, we crawl a large Arabic corpus that is domain specific to religious discourse. Our approach consists of two steps, first we train a BERT model to predict a set of possible answers in a passage…
Recent advances with language models (e.g. BERT, XLNet, ...), have allowed surpassing human performance on complex NLP tasks such as Reading Comprehension. However, labeled datasets for training are available mostly in English which makes it difficult to acknowledge progress in other languages. Fortunately, models are now pre-trained on unlabeled data from hundreds of languages and exhibit interesting transfer abilities from one language to another. In this paper, we show that multilingual BERT is naturally capable of zero-shot transfer for an extractive Question Answering task (eQA) from Engl…
Existing methods for open-retrieval question answering in lower resource languages (LRLs) lag significantly behind English. They not only suffer from the shortcomings of non-English document retrieval, but are reliant on language-specific supervision for either the task or translation. We formulate a task setup more realistic to available resources, that circumvents document retrieval to reliably transfer knowledge from English to lower resource languages. Assuming a strong English question answering model or database, we compare and analyze methods that pivot through English: to map foreign q…
MKG-Rank: Enhancing Large Language Models with Knowledge Graph for Multilingual Medical Question Answering
University of Tokyo（东京大学）；Texas A&M University（德克萨斯农工大学）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Smartor LLC（斯马特有限责任公司）；NEC Laboratories America（美国电气公司实验室）；University of Geneva（日内瓦大学）
University of Copenhagen（哥本哈根大学）；Yale-NUS College（耶鲁-新加坡国大学院）；Technical University of Munich（慕尼黑工业大学）
We study the performance of monolingual and multilingual language models on the task of question-answering (QA) on three diverse languages: English, Finnish and Japanese. We develop models for the tasks of (1) determining if a question is answerable given the context and (2) identifying the answer texts within the context using IOB tagging. Furthermore, we attempt to evaluate the effectiveness of a pre-trained multilingual encoder (Multilingual BERT) on cross-language zero-shot learning for both the answerability and IOB sequence classifiers.
We describe our two-stage system for the Multilingual Information Access (MIA) 2022 Shared Task on Cross-Lingual Open-Retrieval Question Answering. The first stage consists of multilingual passage retrieval with a hybrid dense and sparse retrieval strategy. The second stage consists of a reader which outputs the answer from the top passages returned by the first stage. We show the efficacy of using a multilingual language model with entity representations in pretraining, sparse retrieval signals to help dense retrieval, and Fusion-in-Decoder. On the development set, we obtain 43.46 F1 on XOR-T…
Coupled with the availability of large scale datasets, deep learning architectures have enabled rapid progress on the Question Answering task. However, most of those datasets are in English, and the performances of state-of-the-art multilingual models are significantly lower when evaluated on non-English data. Due to high data collection costs, it is not realistic to obtain annotated data for each language one desires to support. We propose a method to improve the Cross-lingual Question Answering performance without requiring additional annotated data, leveraging Question Generation models to …
Leveraging Synthetic Data for Question Answering with Multilingual LLMs in the Agricultural Domain
CSIR-Central Scientific Instruments Organisation, India（印度CSIR-中央科学仪器组织）；Academy of Scientific and Innovative Research (AcSIR), Ghaziabad, U.P, India（印度科学与创新研究院（AcSIR））；School of Mathematical and Computational Sciences, Massey University, New Zealand（梅西大学数学与计算科学学院）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；IBM India Research Lab（IBM印度研究实验室）
Natural language processing (NLP) tasks (e.g. question-answering in English) benefit from knowledge of other tasks (e.g. named entity recognition in English) and knowledge of other languages (e.g. question-answering in Spanish). Such shared representations are typically learned in isolation, either across tasks or across languages. In this work, we propose a meta-learning approach to learn the interactions between both tasks and languages. We also investigate the role of different sampling strategies used during meta-learning. We present experiments on five different tasks and six different la…
Retrieval based open-domain QA systems use retrieved documents and answer-span selection over retrieved documents to find best-answer candidates. We hypothesize that multilingual Question Answering (QA) systems are prone to information inconsistency when it comes to documents written in different languages, because these documents tend to provide a model with varying information about the same topic. To understand the effects of the biased availability of information and cultural influence, we analyze the behavior of multilingual open-domain question answering models with a focus on retrieval …
Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Peking University（北京大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai University（上海大学）
University of Mannheim（曼海姆大学）；University of Verona（维罗纳大学）；University of Würzburg（维尔茨堡大学）
This paper introduces our proposed system for the MIA Shared Task on Cross-lingual Open-retrieval Question Answering (COQA). In this challenging scenario, given an input question the system has to gather evidence documents from a multilingual pool and generate from them an answer in the language of the question. We devised several approaches combining different model variants for three main components: Data Augmentation, Passage Retrieval, and Answer Generation. For passage retrieval, we evaluated the monolingual BM25 ranker against the ensemble of re-rankers based on multilingual pretrained l…
University of Washington（华盛顿大学）；Google Research（谷歌研究院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Allen Institute for AI（艾伦人工智能研究所）
Multilingual question answering tasks typically assume answers exist in the same language as the question. Yet in practice, many languages face both information scarcity -- where languages have few reference articles -- and information asymmetry -- where questions reference concepts from other cultures. This work extends open-retrieval question answering to a cross-lingual setting enabling questions from one language to be answered via answer content from another language. We construct a large-scale dataset built on questions from TyDi QA lacking same-language answers. Our task formulation, ca…
What Factors Affect LLMs and RLLMs in Financial Question Answering?
School of Computer Science and Engineering, Macau University of Science and Technology, China（澳门科技大学计算机科学与工程学院）；SKLPlanets, Macau University of Science and Technology, China（澳门科技大学月球与行星科学国家重点实验室）；School of Economics, Anhui University, China（安徽大学经济学院）；School of Energy and Power Engineering, Huazhong University of Science and Technology, China（华中科技大学能源与动力工程学院）
Anhalt University of Applied Sciences（安哈尔特应用科学大学）；Laboratoire Hubert Curien CNRS UMR 5516 Université de Lyon（里昂大学于贝尔·居里安实验室（CNRS UMR 5516））；University of Hamburg（汉堡大学）
The ability to have the same experience for different user groups (i.e., accessibility) is one of the most important characteristics of Web-based systems. The same is true for Knowledge Graph Question Answering (KGQA) systems that provide the access to Semantic Web data via natural language interface. While following our research agenda on the multilingual aspect of accessibility of KGQA systems, we identified several ongoing challenges. One of them is the lack of multilingual KGQA benchmarks. In this work, we extend one of the most popular KGQA benchmarks - QALD-9 by introducing high-quality …
Sofia University “St. Kliment Ohridski”（索非亚大学“圣克里门特·奥赫里德斯基”）；Heidelberg University（海德堡大学）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）
We propose EXAMS -- a new benchmark dataset for cross-lingual and multilingual question answering for high school examinations. We collected more than 24,000 high-quality high school exam questions in 16 languages, covering 8 language families and 24 school subjects from Natural Sciences and Social Sciences, among others. EXAMS offers a fine-grained evaluation framework across multiple languages and subjects, which allows precise analysis and comparison of various models. We perform various experiments with existing top-performing multilingual pre-trained models and we show that EXAMS offers m…
University of Southern California（南加州大学）；Information Sciences Institute（信息科学研究所）；Adobe Research（Adobe研究院）
Multilingual models, such as M-BERT and XLM-R, have gained increasing popularity, due to their zero-shot cross-lingual transfer learning capabilities. However, their generalization ability is still inconsistent for typologically diverse languages and across different benchmarks. Recently, meta-learning has garnered attention as a promising technique for enhancing transfer learning under low-resource scenarios: particularly for cross-lingual transfer in Natural Language Understanding (NLU). In this work, we propose X-METRA-ADA, a cross-lingual MEta-TRAnsfer learning ADAptation approach for NLU.…
