分析大语言模型在多轮指令遵循中的多语言能力：以阿拉伯语为例
Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabic
AI总结：
本文利用定制的阿拉伯语版MT-Bench基准和GPT-4统一评估器，考察开源大语言模型在阿拉伯语多轮指令遵循中的表现，发现使用多语言和多轮数据集微调的基础模型及小型模型集成能媲美从头训练或专有模型。
摘要：
尽管在跨各种任务对大语言模型（LLMs）进行基准测试方面已取得显著进展，但对其在阿拉伯语等较少测试语言中响应多轮指令的能力仍缺乏全面评估。本文详细考察了开源LLMs在阿拉伯语此类场景中的熟练度。利用定制的MT-Bench基准套件阿拉伯语翻译版本，我们采用GPT-4作为英语和阿拉伯语查询的统一评估器，以评估并比较LLMs在各类开放式任务上的表现。我们的发现揭示了在用英语或阿拉伯语进行指令时，模型在不同任务类别（如逻辑与文学）上的响应存在差异。我们发现，使用多语言和多轮数据集微调的基础模型，可以与在多语言数据上从头训练的模型相媲美。最后，我们假设由小型开源LLMs组成的集成模型可以在该基准上与专有LLMs表现相当。
英文摘要：
While significant progress has been made in benchmarking Large Language Models (LLMs) across various tasks, there is a lack of comprehensive evaluation of their abilities in responding to multi-turn instructions in less-commonly tested languages like Arabic. Our paper offers a detailed examination of the proficiency of open LLMs in such scenarios in Arabic. Utilizing a customized Arabic translation of the MT-Bench benchmark suite, we employ GPT-4 as a uniform evaluator for both English and Arabic queries to assess and compare the performance of the LLMs on various open-ended tasks. Our findings reveal variations in model responses on different task categories, e.g., logic vs. literacy, when instructed in English or Arabic. We find that fine-tuned base models using multilingual and multi-turn datasets could be competitive to models trained from scratch on multilingual data. Finally, we hypothesize that an ensemble of small, open LLMs could perform competitively to proprietary LLMs on the benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
