LINGOLY：面向低资源与灭绝语言的奥赛级语言推理谜题基准
LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages
AI总结：
该研究提出面向低资源与灭绝语言的奥赛级语言推理谜题基准LingOly，覆盖90余种语言、千余道多难度题型，评测发现当前大语言模型在排除记忆的多步域外推理上仍存在明显短板。
摘要：
本文提出LingOly基准，这是一种用于评测大语言模型高级推理能力的新型基准。我们借助高难度的语言学奥赛谜题，评估两大能力：（i）在极低资源或灭绝语言中进行上下文内语言模式识别与泛化的能力；（ii）遵循复杂任务指令的能力。LingOly基准覆盖90余种以低资源语言为主的语种，最大程度降低了数据污染问题，包含1133道题目，涵盖6种题型和5个人类难度等级。我们同时采用直接准确率和与无上下文基线对比的方式评估性能，以惩罚记忆行为。11个先进大语言模型的得分表明该基准具有挑战性，模型在高难度题目上表现不佳。在更难的题目上，即便顶级模型也仅达到38.7%的准确率，仅比无上下文基线提升24.7%。大型闭源模型通常优于开源模型，且总体而言，语言的资源越丰富，得分越高。这些结果表明，在排除记忆因素的情况下，真正的多步域外推理对当前语言模型而言仍是一项挑战。
英文摘要：
In this paper, we present the LingOly benchmark, a novel benchmark for advanced reasoning abilities in large language models. Using challenging Linguistic Olympiad puzzles, we evaluate (i) capabilities for in-context identification and generalisation of linguistic patterns in very low-resource or extinct languages, and (ii) abilities to follow complex task instructions. The LingOly benchmark covers more than 90 mostly low-resource languages, minimising issues of data contamination, and contains 1,133 problems across 6 formats and 5 levels of human difficulty. We assess performance with both direct accuracy and comparison to a no-context baseline to penalise memorisation. Scores from 11 state-of-the-art LLMs demonstrate the benchmark to be challenging, and models perform poorly on the higher difficulty problems. On harder problems, even the top model only achieved 38.7% accuracy, a 24.7% improvement over the no-context baseline. Large closed models typically outperform open models, and in general, the higher resource the language, the better the scores. These results indicate, in absence of memorisation, true multi-step out-of-domain reasoning remains a challenge for current language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
