In this paper, we compare general-purpose models, GPT-4-Turbo and Llama-3-8b, with special-purpose models--XLM-Roberta-large, mT5-large, and Llama-3-8b--that have been fine-tuned on specific tasks. We focus on seven classification and seven generation tasks to evaluate the performance of these models on Urdu language. Urdu has 70 million native speakers, yet it remains underrepresented in Natural Language Processing (NLP). Despite the frequent advancements in Large Language Models (LLMs), their performance in low-resource languages, including Urdu, still needs to be explored. We also conduct a human evaluation for the generation tasks and compare the results with the evaluations performed by GPT-4-Turbo, Llama-3-8b and Claude 3.5 Sonnet. We find that special-purpose models consistently outperform general-purpose models across various tasks. We also find that the evaluation done by GPT-4-Turbo for generation tasks aligns more closely with human evaluation compared to the evaluation the evaluation done by Llama-3-8b. This paper contributes to the NLP community by providing insights into the effectiveness of general and specific-purpose LLMs for low-resource languages.
Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）；Cohere；Tencent AI Lab（腾讯AI实验室）
GSCo: Towards Generalizable AI in Medicine via Generalist-Specialist Collaboration
The Hong Kong University of Science and Technology（香港科技大学）；Harvard University（哈佛大学）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；Shenzhen People’s Hospital（深圳市人民医院）；Weill（威尔（未完整显示））
TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
Mohamed bin Zayed University of Artificial Intelligence（莫莫德·本·扎耶德人工智能大学）；Monash University（墨尔本大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Southampton（南安普顿大学）
Belief Coevolution in a Social Network of Generalist and Specialist Large Language Models: A Controlled Study
Khoury College of Computer Sciences, Northeastern University（东北大学库尔计算机科学学院）；Network Science Institute, Northeastern University（东北大学网络科学研究所）；Santa Fe Institute（圣塔菲研究所）
RareBench: Can LLMs Serve as Rare Diseases Specialists?
Tsinghua University（清华大学）；Peking Union Medical College Hospital（北京协和医院）；Chinese Academy of Medical Sciences（中国医学科学院）；Peking Union Medical College（北京协和医学院）
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
University of California, San Diego（加州大学圣地亚哥分校）；ByteDance（字节跳动）；University of California, Merced（加州大学默塞德分校）；University of Southern California（南加州大学）；University at Buffalo（布法罗大学）；The University of Queensland（昆士兰大学）
GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training
Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
