MLissard：多语言长序列与简单序列推理基准
MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks
AI总结：
该研究提出多语言推理基准MLissard以评估模型处理不同长度文本的能力，发现序列复杂度提升会导致各模型多语言性能下降，且非英语上下文示例可显著提升外推性能。
摘要：
当前语言模型已能够解决需要处理包含数十万token的长序列的任务，但它们在需要重复运用简单规则的任务上常常失败，即便序列长度远短于训练时见过的序列。例如，最先进的大语言模型（LLM）能找出两个最多含20个项的列表中的共同项，但当列表有80个项时就会失败。本文提出MLissard，这是一个多语言基准，旨在评估模型处理和生成不同长度文本的能力，并提供了一种控制序列复杂度的机制。\n我们对开源模型和专有模型的评估显示，随着序列复杂度提升，所有模型在所有语言上的性能均持续下降。令人意外的是，使用英语以外语言的上下文示例能显著提升外推性能。数据集与代码可在https://github.com/unicamp-dl/Lissard 获取。
英文摘要：
Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models' abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly. The datasets and code are available at https://github.com/unicamp-dl/Lissard
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
