The first step in any NLP pipeline is to split the text into individual tokens. The most obvious and straightforward approach is to use words as tokens. However, given a large text corpus, representing all the words is not efficient in terms of vocabulary size. In the literature, many tokenization algorithms have emerged to tackle this problem by creating subwords which in turn limits the vocabulary size in a given text corpus. Most tokenization techniques are language-agnostic i.e they don't incorporate the linguistic features of a given language. Not to mention the difficulty of evaluating such techniques in practice. In this paper, we introduce three new tokenization algorithms for Arabic and compare them to three other baselines using unsupervised evaluations. In addition to that, we compare all the six algorithms by evaluating them on three supervised classification tasks which are sentiment analysis, news classification and poetry classification using six publicly available datasets. Our experiments show that none of the tokenization technique is the best choice overall and that the performance of a given tokenization algorithm depends on the size of the dataset, type of the task, and the amount of morphology that exists in the dataset. However, some tokenization techniques are better overall as compared to others on various text classification tasks.
Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMs
Saudi Data & AI Authority (SDAIA)(沙特数据与人工智能局);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学);Princess Nourah Bint Abdulrahman University (PNU)(普罗瑟·努拉·本·阿卜杜勒拉赫曼大学)
Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
University of Melbourne(墨尔本大学);National University of Mongolia(蒙古国立大学);University of Edinburgh(爱丁堡大学);Ben-Gurion University of the Negev(内盖夫本-古里安大学);IMT Atlantique(IMT大西洋电信学院)
Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models
King Fahd University of Petroleum and Minerals(法赫德国王石油与矿业大学);ARBML;SDAIA-KFUPM Joint Research Center for Artificial Intelligence(沙特数据与人工智能局-法赫德国王石油与矿业大学人工智能联合研究中心);Volvo Cars R&D Tech Center(沃尔沃汽车研发技术中心)
AraReasoner: Evaluating Reasoning-Based LLMs for Arabic NLP
King Fahd University of Petroleum and Minerals(国王法赫德石油和矿物大学);SDAIA–KFUPM Joint Research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心);Mohammed VI Polytechnic University(穆莱·易卜拉欣(polytechnic)大学)
Towards stable AI systems for Evaluating Arabic Pronunciations
Bio-Medical Imaging Core, Core Technology Platforms, New York University Abu Dhabi(生物医学成像核心、核心技术平台、纽约大学阿布扎比分校);Institut de recherche technologique SystemX(SystemX技术研究所);NYUAD Research Institute, New York University Abu Dhabi(NYUAD研究学院、纽约大学阿布扎比分校);Courant Institute of Mathematical Sciences, New York University(Courant数学科学研究所、纽约大学)
MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Technical University of Berlin(柏林工业大学)
Splintering Nonconcatenative Languages for Better Tokenization
Department of Computer Science(计算机科学系);Data Science Research Center(数据科学研究中心);Bar-Ilan University(巴伊兰大学);Ben-Gurion University of the Negev(贝内-约瑟夫大学(内盖夫分校));δ DICTA, Jerusalem, Israel(δ DICTA,耶路撒冷,以色列)
From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
Department of Computer Science, King Khalid University(国王 Khalid 大学计算机科学系);Department of Computer Science and Artificial Intelligence, Umm Al-Qura University(乌姆·阿勒·库拉大学计算机科学与人工智能系);Imam Abdulrahman Bin Faisal University(伊玛目阿卜杜勒拉赫曼·本·法伊萨尔大学)
Google Translate Error Analysis for Mental Healthcare Information: Evaluating Accuracy, Comprehensibility, and Implications for Multilingual Healthcare Communication
University of Surrey(萨里大学);University of Amsterdam(阿姆斯特丹大学)
Texas A&M University(德克萨斯农工大学);Qatar Computing Research Institute(卡塔尔计算研究所);Birzeit University(比尔宰特大学);Hamad Bin Khalifa University(哈马德·本·哈利法大学);University of Toronto(多伦多大学)
Evaluating Text Style Transfer: A Nine-Language Benchmark for Text Detoxification
AIRI;Universidade de Santiago de Compostela(圣塞巴斯蒂安大学);Technical University of Munich (TUM)(慕尼黑技术大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Skoltech