代码混合还是不混合：衡量多语言LLM代码混合能力的新方法
Code-Mixer Ya Nahi: Novel Approaches to Measuring Multilingual LLMs' Code-Mixing Capabilities
AI总结：
本文提出Rule-Based Prompting提示技术，评估了GPT-3.5-turbo、GPT-4和Gemini Pro在五个语言对上的代码混合机器翻译能力，并构建了黄金标准数据集和代码混合聊天机器人。
摘要：
多语言大型语言模型（LLMs）在机器翻译（MT）任务中展现了卓越的性能。然而，它们在语码转换（即在话语中混合两种或多种语言的做法）背景下的MT能力仍未被充分探索。本文介绍了Rule-Based Prompting，一种用于生成代码混合句子的新型提示技术。我们使用$k$-shot prompting（$k\in\{0, 1, 10, 20\}$）和Rule-Based Prompting，测量并比较了3种流行多语言LLM的代码混合MT能力：GPT-3.5-turbo、GPT-4和Gemini Pro，涵盖五个语言对：英语-{印地语、孟加拉语、古吉拉特语、法语、西班牙语}。研究结果表明，尽管$k$-shot prompting通常能取得最佳结果，但Rule-Based Prompting在生成代码混合风格各异的独特代码混合句子方面显示出潜力。我们还使用$k$-shot prompting来评估多语言LLM从代码混合到英语的翻译能力。为此，我们创建了一个涵盖五个语言对的黄金标准代码混合数据集：英语-{印地语、孟加拉语、古吉拉特语、法语、西班牙语}。作为我们工作的实际应用，我们创建了一个代码混合聊天机器人。
英文摘要：
Multilingual Large Language Models (LLMs) have demonstrated exceptional performance in Machine Translation (MT) tasks. However, their MT abilities in the context of code-switching (the practice of mixing two or more languages in an utterance) remain under-explored. In this paper, we introduce Rule-Based Prompting, a novel prompting technique to generate code-mixed sentences. We measure and compare the code-mixed MT abilities of 3 popular multilingual LLMs: GPT-3.5-turbo, GPT-4, and Gemini Pro across five language pairs: English-{Hindi, Bengali, Gujarati, French, Spanish} using $k$-shot prompting ($k\in\{0, 1, 10, 20\}$) and Rule-Based Prompting. Our findings suggest that though $k$-shot prompting often leads to the best results, Rule-Based prompting shows promise in generating unique code-mixed sentences that vary in their style of code-mixing. We also use $k$-shot prompting to gauge the code-mixed to English translation abilities of multilingual LLMs. For this purpose, we create a gold-standard code-mixed dataset spanning five language pairs: English-{Hindi, Bengali, Gujarati, French, Spanish}. As a real-world application of our work, we create a code-mixed chatbot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
