SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models
Compumacy for Artificial Intelligence Solutions（Compumacy人工智能解决方案公司）；Taibah University（泰巴大学）；George Mason University（乔治梅森大学）；Queen’s University Belfast（贝尔法斯特女王大学）
Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh, Ihsen Alouani, Mohammed E. Fouda
While different stakeholders are trying to leverage Arabic Language Models (ALMs), safety alignment in ALMs remains largely underexplored, hindering their mainstream adoption. Existing safety benchmarks are predominantly English-centric and evaluate Arabic only in its standardized form, obscuring fine-grained safety vulnerabilities in Arabic NLP systems. This paper introduces SalamahBench, a unified benchmark of 8{,}270 human-verified harmful prompts across ML Commons hazard categories, each rendered in Modern Standard Arabic (MSA) and five regional Arabic varieties, namely Egyptian, Syrian, Saudi, Lebanese, and Moroccan, for a total of 49{,}620 paired instances. To analyze the resulting data, we introduce two complementary metrics, namely Dialect Shift, which measures a model's aggregate change in safety under dialectal reformulation, and Category-Specific Dialect Deviation, which isolates harm categories whose change departs from that aggregate trend. Evaluating models such as Fanar 2, ALLaM 2, and Karnak 1 under multiple safeguard configurations, we find that cross-variety robustness is strongly model dependent, and that aggregate scores can conceal category-level divergence. Our findings highlight the necessity of evaluating Arabic model safety jointly across linguistic varieties and harm domains rather than relying on aggregate scores or MSA alone.
DialectSentEval 2026: Arabic Dialect Sentiment Analysis and Swapping Shared Task
King Fahd University of Petroleum and Minerals（法赫德国王石油矿产大学）；Onaizah Colleges（欧奈扎学院）；University College of Applied Sciences（应用科学大学学院）；Lancaster University（兰卡斯特大学）；UM6P（穆罕默德六世理工大学）；VinUniversity；El Technology；University of Luxembourg（卢森堡大学）
AHaSIS: Shared Task on Sentiment Analysis for Arabic Dialects
School of Computing and Communications, Lancaster University, UK（计算与通信学院，兰卡斯特大学，英国）；Jazan University, Saudi Arabia（朱兹安大学，沙特阿拉伯）；Mohammed VI Polytechnic University, Morocco（穆莱·易卜拉欣·聚technic大学，摩洛哥）；King Fahd University of Petroleum and Minerals, Saudi Arabia（国王 Fahd石油和矿物大学，沙特阿拉伯）
The University of British Columbia（不列颠哥伦比亚大学）；Carnegie Mellon University in Qatar（卡塔尔卡内基梅隆大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）
We present the results and findings of the First Nuanced Arabic Dialect Identification Shared Task (NADI). This Shared Task includes two subtasks: country-level dialect identification (Subtask 1) and province-level sub-dialect identification (Subtask 2). The data for the shared task covers a total of 100 provinces from 21 Arab countries and are collected from the Twitter domain. As such, NADI is the first shared task to target naturally-occurring fine-grained dialectal text at the sub-country level. A total of 61 teams from 25 countries registered to participate in the tasks, thus reflecting t…
AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation
The University of British Columbia（不列颠哥伦比亚大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油矿产大学）；Lancaster University（兰卡斯特大学）；VinUniversity；Hamad Bin Khalifa University（哈马德·本·哈利法大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task
The University of British Columbia（不列颠哥伦比亚大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；The University of Edinburgh（爱丁堡大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）；Carnegie Mellon University in Qatar（卡塔尔卡内基梅隆大学）
Exploiting Dialect Identification in Automatic Dialectal Text Normalization
Computational Approaches to Modeling Language Lab（语言建模计算方法实验室）；New York University Abu Dhabi（纽约大学迪拜分校）；Carnegie Mellon University in Qatar（卡内基梅隆大学多哈分校）
Almieyar: A Culturally Grounded Benchmark for Multi-Dialect Arabic Speech Recognition
QCRI, HBKU（卡塔尔计算研究所，哈马德·本·哈利法大学）；Qatar University（卡塔尔大学）；UDST（多哈科技大学）；Algo AI；UCL（伦敦大学学院）；University of Tripoli（的黎波里大学）；AUC（开罗美国大学）；KAUST（阿卜杜拉国王科技大学）；CMU-Q（卡内基梅隆大学卡塔尔分校）；KFUPM（法赫德国王石油与矿产大学）；Alfaisal University（费萨尔大学）；Damascus University（大马士革大学）；Princeton University（普林斯顿大学）；ENSIAS, Mohammed V University（穆罕默德五世大学国家计算机科学与系统分析学院）；Sultan Qaboos University（苏丹卡布斯大学）；DFKI（德国人工智能研究中心）；University of Waterloo（滑铁卢大学）；USTHB（阿尔及尔科技大学）
This paper presents our approach to address the EACL WANLP-2021 Shared Task 1: Nuanced Arabic Dialect Identification (NADI). The task is aimed at developing a system that identifies the geographical location(country/province) from where an Arabic tweet in the form of modern standard Arabic or dialect comes from. We solve the task in two parts. The first part involves pre-processing the provided dataset by cleaning, adding and segmenting various parts of the text. This is followed by carrying out experiments with different versions of two Transformer based models, AraBERT and AraELECTRA. Our fi…
In this paper, we tackle the Nuanced Arabic Dialect Identification (NADI) shared task (Abdul-Mageed et al., 2021) and demonstrate state-of-the-art results on all of its four subtasks. Tasks are to identify the geographic origin of short Dialectal (DA) and Modern Standard Arabic (MSA) utterances at the levels of both country and province. Our final model is an ensemble of variants built on top of MARBERT that achieves an F1-score of 34.03% for DA at the country-level development set -- an improvement of 7.63% from previous work.
The University of British Columbia（不列颠哥伦比亚大学）；Carnegie Mellon University in Qatar（卡内基梅隆大学卡塔尔分校）；New York University Abu Dhabi（纽约大学阿布扎比分校）
We describe findings of the third Nuanced Arabic Dialect Identification Shared Task (NADI 2022). NADI aims at advancing state of the art Arabic NLP, including on Arabic dialects. It does so by affording diverse datasets and modeling opportunities in a standardized context where meaningful comparisons between models and approaches are possible. NADI 2022 targeted both dialect identification (Subtask 1) and dialectal sentiment analysis (Subtask 2) at the country level. A total of 41 unique teams registered for the shared task, of whom 21 teams have actually participated (with 105 valid submissio…
Munsit at NADI 2025 Shared Task 2: Pushing the Boundaries of Multidialectal Arabic ASR with Weakly Supervised Pretraining and Continual Supervised Fine-tuning
The University of British Columbia（不列颠哥伦比亚大学）；Carnegie Mellon University in Qatar（卡内基梅隆大学卡塔尔分校）；New York University Abu Dhabi（纽约大学阿布扎比分校）
We present the findings and results of the Second Nuanced Arabic Dialect Identification Shared Task (NADI 2021). This Shared Task includes four subtasks: country-level Modern Standard Arabic (MSA) identification (Subtask 1.1), country-level dialect identification (Subtask 1.2), province-level MSA identification (Subtask 2.1), and province-level sub-dialect identification (Subtask 2.2). The shared task dataset covers a total of 100 provinces from 21 Arab countries, collected from the Twitter domain. A total of 53 teams from 23 countries registered to participate in the tasks, thus reflecting th…
Qatar Computing Research Institute（卡塔尔计算研究所）；Hamad Bin Khalifa University (HBKU)（哈马德·本·哈利法大学（HBKU））
We present QADI, an automatically collected dataset of tweets belonging to a wide range of country-level Arabic dialects -covering 18 different countries in the Middle East and North Africa region. Our method for building this dataset relies on applying multiple filters to identify users who belong to different countries based on their account descriptions and to eliminate tweets that are either written in Modern Standard Arabic or contain inappropriate language. The resultant dataset contains 540k tweets from 2,525 users who are evenly distributed across 18 Arab countries. Using intrinsic eva…
ARCADE: A City-Scale Corpus for Fine-Grained Arabic Dialect Tagging
Tuwaiq Academy（图瓦伊克学院）；Robotics and Internet-of-Things Laboratory（机器人与物联网实验室）；Prince Sultan University（普森大学）；Higher School of Communication of Tunis (SUP’COM)（突尼斯通信高级学校）
NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task
The University of British Columbia（不列颠哥伦比亚大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University in Qatar（卡内基梅隆大学卡塔尔分校）；New York University Abu Dhabi（纽约大学阿布扎比分校）
The University of British Columbia（不列颠哥伦比亚大学）；American University of Beirut（贝鲁特美国大学）；Concordia University（康考迪亚大学）；Arab Academy for Science and Technology（阿拉伯科学与技术学院）
Word embeddings are a core component of modern natural language processing systems, making the ability to thoroughly evaluate them a vital task. We describe DiaLex, a benchmark for intrinsic evaluation of dialectal Arabic word embedding. DiaLex covers five important Arabic dialects: Algerian, Egyptian, Lebanese, Syrian, and Tunisian. Across these dialects, DiaLex provides a testbank for six syntactic and semantic relations, namely male to female, singular to dual, singular to plural, antonym, comparative, and genitive to past tense. DiaLex thus consists of a collection of word pairs representi…
College of Humanities and Social Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学人文与社会科学学院）；Carnegie Mellon University Qatar（卡内基梅隆大学卡塔尔分校）
In this paper, we present the annotation pipeline and the guidelines we wrote as part of an effort to create a large manually annotated Arabic author profiling dataset from various social media sources covering 16 Arabic countries and 11 dialectal regions. The target size of the annotated ARAP-Tweet corpus is more than 2.4 million words. We illustrate and summarize our general and dialect-specific guidelines for each of the dialectal regions selected. We also present the annotation framework and logistics. We control the annotation quality frequently by computing the inter-annotator agreement …
College of Humanities and Social Sciences, Hamad Bin Khalifa University（哈马德·本·哈利法大学人文与社会科学学院）；Carnegie Mellon University Qatar（卡内基梅隆大学卡塔尔分校）
In this paper, we present Arap-Tweet, which is a large-scale and multi-dialectal corpus of Tweets from 11 regions and 16 countries in the Arab world representing the major Arabic dialectal varieties. To build this corpus, we collected data from Twitter and we provided a team of experienced annotators with annotation guidelines that they used to annotate the corpus for age categories, gender, and dialectal variety. During the data collection effort, we based our search on distinctive keywords that are specific to the different Arabic dialects and we also validated the location using Twitter API…
New York University Abu Dhabi（纽约大学阿布扎比分校）；University of Bahrain（巴林大学）
Most Arabic natural language processing tools and resources are developed to serve Modern Standard Arabic (MSA), which is the official written language in the Arab World. Some Dialectal Arabic varieties, notably Egyptian Arabic, have received some attention lately and have a growing collection of resources that include annotated corpora and morphological analyzers and taggers. Gulf Arabic, however, lags behind in that respect. In this paper, we present the Gumar Corpus, a large-scale corpus of Gulf Arabic consisting of 110 million words from 1,200 forum novels. We annotate the corpus for sub-d…
AraFinNLP 2024: The First Arabic Financial NLP Shared Task
Birzeit University（比尔宰特大学）；Lancaster University（兰卡斯特大学）；King Saud University（沙特国王大学）；Mohammed VI Polytechnic University（穆罕默德六世理工大学）；Carnegie Mellon University（卡内基梅隆大学）
We report our models for detecting age, language variety, and gender from social media data in the context of the Arabic author profiling and deception detection shared task (APDA). We build simple models based on pre-trained bidirectional encoders from transformers (BERT). We first fine-tune the pre-trained BERT model on each of the three datasets with shared task released data. Then we augment shared task data with in-house data for gender and dialect, showing the utility of augmenting training data. Our best models on the shared task test data are acquired with a majority voting of various …
Morphological tagging is challenging for morphologically rich languages due to the large target space and the need for more training data to minimize model sparsity. Dialectal variants of morphologically rich languages suffer more as they tend to be more noisy and have less resources. In this paper we explore the use of multitask learning and adversarial training to address morphological richness and dialectal variations in the context of full morphological tagging. We use multitask learning for joint morphological modeling for the features within two dialects, and as a knowledge-transfer sche…
LIG（信息学实验室）；University of Rome Sapienza（罗马大学（萨皮恩扎））
This article describes the constitution process of the first morpho-syntactically annotated Tunisian Arabish Corpus (TArC). Arabish, also known as Arabizi, is a spontaneous coding of Arabic dialects in Latin characters and arithmographs (numbers used as letters). This code-system was developed by Arabic-speaking users of social media in order to facilitate the writing in the Computer-Mediated Communication (CMC) and text messaging informal frameworks. There is variety in the realization of Arabish amongst dialects, and each Arabish code-system is under-resourced, in the same way as most of the…
Northwest University（西北大学）；University of Essex（埃塞克斯大学）
Arabic sentiment analysis has become an important research field in recent years. Initially, work focused on Modern Standard Arabic (MSA), which is the most widely-used form. Since then, work has been carried out on several different dialects, including Egyptian, Levantine and Moroccan. Moreover, a number of datasets have been created to support such work. However, up until now, less work has been carried out on Sudanese Arabic, a dialect which has 32 million speakers. In this paper, two new publicly available datasets are introduced, the 2-Class Sudanese Sentiment Dataset (SudSenti2) and the …
ICOSI Laboratory, University of Khenchela（肯谢拉大学ICOSI实验室）；LIRE Laboratory, University of Constantine 2（君士坦丁第二大学LIRE实验室）；TechNE Laboratory, University of Poitiers（普瓦捷大学TechNE实验室）
In this paper, we propose our enhanced approach to create a dedicated corpus for Algerian Arabic newspapers comments. The developed approach has to enhance an existing approach by the enrichment of the available corpus and the inclusion of the annotation step by following the Model Annotate Train Test Evaluate Revise (MATTER) approach. A corpus is created by collecting comments from web sites of three well know Algerian newspapers. Three classifiers, support vector machines, na{ï}ve Bayes, and k-nearest neighbors, were used for classification of comments into positive and negative classes. To …
Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；Kanari AI（卡纳里人工智能公司）
We introduce the largest transcribed Arabic speech corpus, QASR, collected from the broadcast domain. This multi-dialect speech dataset contains 2,000 hours of speech sampled at 16kHz crawled from Aljazeera news channel. The dataset is released with lightly supervised transcriptions, aligned with the audio segments. Unlike previous datasets, QASR contains linguistically motivated segmentation, punctuation, speaker information among others. QASR is suitable for training and evaluating speech recognition systems, acoustics- and/or linguistics- based Arabic dialect identification, punctuation res…
