在巴别中摸索：对 ChatGPT 语言识别能力的调查
Fumbling in Babel: An Investigation into ChatGPT's Language Identification Ability
AI总结：
本文构建涵盖670种语言的 Babel-670 基准，在多种提示设置下评估 GPT-3.5 与 GPT-4 的语言识别能力，发现其落后于小型微调工具，尤其在非洲语言上表现不佳。
摘要：
ChatGPT 最近已成为一种强大的 NLP 工具，能够执行多种任务。然而，ChatGPT 能够处理的语言范围在很大程度上仍是一个谜。为揭示 ChatGPT“懂”哪些语言，我们调查了其语言识别（LID）能力。为此，我们编制了 Babel-670，这一基准包含 670 种语言，代表五大洲使用的 24 个语系。Babel-670 中的语言涵盖从极高资源到极低资源的完整范围。随后，我们研究 ChatGPT（包括 GPT-3.5 和 GPT-4）在以下方面的能力：（i）识别语言名称和语言代码；（ii）在零样本和少样本条件下；（iii）提供与不提供标签集的情况。与规模较小、经过微调的 LID 工具相比，我们发现 ChatGPT 表现落后。例如，它在非洲语言上性能较差。我们得出结论：当前的大型语言模型需要进一步发展，才能充分服务于多元社群。
英文摘要：
ChatGPT has recently emerged as a powerful NLP tool that can carry out a variety of tasks. However, the range of languages ChatGPT can handle remains largely a mystery. To uncover which languages ChatGPT `knows', we investigate its language identification (LID) abilities. For this purpose, we compile Babel-670, a benchmark comprising 670 languages representing 24 language families spoken in five continents. Languages in Babel-670 run the gamut from the very high-resource to the very low-resource. We then study ChatGPT's (both GPT-3.5 and GPT-4) ability to (i) identify language names and language codes (ii) under zero- and few-shot conditions (iii) with and without provision of a label set. When compared to smaller finetuned LID tools, we find that ChatGPT lags behind. For example, it has poor performance on African languages. We conclude that current large language models would benefit from further development before they can sufficiently serve diverse communities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
