Multi-IF：在多轮与多语言指令遵循上对LLMs进行基准测试
Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
AI总结：
本文提出Multi-IF基准，通过结合LLM与人工标注构建包含4501个多语言多轮对话的数据集，评估14个先进模型，发现模型在多轮交互和非拉丁语种中指令遵循能力显著下降。
摘要：
大型语言模型（LLMs）在包括指令遵循在内的各种任务中展现出了令人印象深刻的能力，这对于使模型输出与用户期望保持一致至关重要。然而，由于人类语言的复杂性和主观性，评估LLMs遵循指令的能力仍然具有挑战性。当前的基准测试主要集中在单轮、单语言指令上，这不能充分反映需要处理多轮和多语言交互的现实世界应用的复杂性。为了弥补这一差距，我们引入了Multi-IF，这是一个旨在评估LLMs在遵循多轮和多语言指令方面熟练程度的新基准。Multi-IF利用结合了LLM和人工标注员的混合框架，通过纳入多轮序列并将英文提示词翻译成另外7种语言来扩展IFEval，最终形成了一个包含4,501个多语言对话的数据集，每个对话有三轮。我们在Multi-IF上对14个最先进的LLMs进行评估，结果显示它比现有的基准测试提出了一个更具挑战性的任务。所有测试的模型在正确执行指令方面，随着轮次的增加，失败率都会升高。例如，o1-preview在所有语言的平均准确率上，从第一轮的0.877下降到第三轮的0.707。此外，使用非拉丁字母的语言（印地语、俄语和中文）通常表现出更高的错误率，这表明模型在多语言能力方面存在潜在局限。我们发布了Multi-IF的提示词和评估代码库，以鼓励在这一关键领域进行进一步研究。
英文摘要：
Large Language Models (LLMs) have demonstrated impressive capabilities in various tasks, including instruction following, which is crucial for aligning model outputs with user expectations. However, evaluating LLMs' ability to follow instructions remains challenging due to the complexity and subjectivity of human language. Current benchmarks primarily focus on single-turn, monolingual instructions, which do not adequately reflect the complexities of real-world applications that require handling multi-turn and multilingual interactions. To address this gap, we introduce Multi-IF, a new benchmark designed to assess LLMs' proficiency in following multi-turn and multilingual instructions. Multi-IF, which utilizes a hybrid framework combining LLM and human annotators, expands upon the IFEval by incorporating multi-turn sequences and translating the English prompts into another 7 languages, resulting in a dataset of 4,501 multilingual conversations, where each has three turns. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. All the models tested showed a higher rate of failure in executing instructions correctly with each additional turn. For example, o1-preview drops from 0.877 at the first turn to 0.707 at the third turn in terms of average accuracy over all languages. Moreover, languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models' multilingual capabilities. We release Multi-IF prompts and the evaluation code base to encourage further research in this critical area.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
