This paper explores the efficacy of large language models (LLMs) for Persian. While ChatGPT and consequent LLMs have shown remarkable performance in English, their efficiency for more low-resource languages remains an open question. We present the first comprehensive benchmarking study of LLMs across diverse Persian language tasks. Our primary focus is on GPT-3.5-turbo, but we also include GPT-4 and OpenChat-3.5 to provide a more holistic evaluation. Our assessment encompasses a diverse set of tasks categorized into classic, reasoning, and knowledge-based domains. To enable a thorough comparison, we evaluate LLMs against existing task-specific fine-tuned models. Given the limited availability of Persian datasets for reasoning tasks, we introduce two new benchmarks: one based on elementary school math questions and another derived from the entrance exams for 7th and 10th grades. Our findings reveal that while LLMs, especially GPT-4, excel in tasks requiring reasoning abilities and a broad understanding of general knowledge, they often lag behind smaller pre-trained models fine-tuned specifically for particular tasks. Additionally, we observe improved performance when test sets are translated to English before inputting them into GPT-3.5. These results highlight the significant potential for enhancing LLM performance in the Persian language. This is particularly noteworthy due to the unique attributes of Persian, including its distinct alphabet and writing styles.
Benchmarking Open-Source Large Language Models for Persian in Zero-Shot and Few-Shot Learning
Department of Medical Engineering（医学工程系）；Electrical Engineering Department（电气工程系）；Sharif University of Technology（谢里夫技术大学）；Department of Electrical Engineering（电气工程系）；University of Tehran（德黑兰大学）
PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language
School of Computer Science Institute for Research in Fundamental Sciences（计算机科学学院基础科学研究研究所）；School of Electrical and Computer Engineering University of Tehran（电气与计算机工程学院塔里班大学）；Department of Computer Science Faculty of Mathematical Sciences Alzahra University（计算机科学系数学科学学院阿扎赫拉大学）
Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology
University Hospital Erlangen（埃尔兰根大学医院）；Comprehensive Cancer Center Erlangen-EMN（埃尔兰根-EMN综合癌症中心）；Pattern Recognition Lab（模式识别实验室）
BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining
Department of Computer Science, Faculty of Education, University of Kufa（计算机科学系，教育学院，科菲大学）；Department of Computer Science, Florida State University（计算机科学系，佛罗里达州立大学）；Department of Computer Engineering, Velayat University（计算机工程系，瓦莱yat大学）；Department of Mechanical Engineering, Amirkabir University of Technology（机械工程系，阿米卡比尔技术大学）；Department of Computer Engineering, Ferdowsi University of Mashhad（计算机工程系，法尔德大学）
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
The Chinese University of Hong Kong（香港中文大学）；Tianjin Medical University（天津医科大学）；Tencent AI Lab（腾讯AI实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University（浙江大学-蚂蚁集团知识图谱联合研究中心）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Alibaba Group（阿里巴巴集团）
DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4
The University of Georgia（佐治亚大学）；Lehigh University（莱恩大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院）；Mayo Clinic（梅奥诊所）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）
Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study
School of Mathematics and Computer Science, Zhejiang Agriculture and Forestry University（浙江农林大学数学与计算机科学学院）；School of Earth Sciences, Zhejiang University（浙江大学地球科学学院）；Financial Big Data Research Institute, Sunyard Technology Co., Ltd.（星宇科技股份有限公司金融大数据研究院）
