MORABLES：一个用寓言评估 LLMs 抽象道德推理的基准
MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables
AI总结：
本文提出了基于历史寓言的基准 MORABLES，通过多项选择和对抗性变体评估 LLMs 的抽象道德推理能力，发现大模型虽表现更好但仍易受对抗操纵，且性能提升主要依赖规模而非推理能力。
摘要：
随着 LLMs 在标准阅读理解基准上表现优异，注意力正转向评估其复杂抽象推理与推断的能力。基于文献的基准具有丰富的叙事和道德深度，为评估此类深层理解能力提供了引人注目的框架。在此，我们提出了 MORABLES，这是一个由历史文学中的寓言和短篇故事构建的、经人工验证的基准。主要任务被构建为针对道德推断的多项选择题，并配有精心设计的干扰项，以挑战模型超越浅层、抽取式的问答。为了进一步压力测试模型的鲁棒性，我们引入了对抗性变体，旨在暴露由于数据污染等问题导致的 LLM 漏洞和捷径。我们的研究结果表明，虽然较大的模型优于较小的模型，但它们仍然容易受到对抗性操纵，并且经常依赖表面模式而非真正的道德推理。这种脆弱性导致了严重的自我矛盾，最好的模型在约 20% 的情况下会根据道德选择的表述方式推翻自己的答案。有趣的是，推理增强型模型未能弥合这一差距，这表明规模——而非推理能力——是性能的主要驱动力。
英文摘要：
As LLMs excel on standard reading comprehension benchmarks, attention is shifting toward evaluating their capacity for complex abstract reasoning and inference. Literature-based benchmarks, with their rich narrative and moral depth, provide a compelling framework for evaluating such deeper comprehension skills. Here, we present MORABLES, a human-verified benchmark built from fables and short stories drawn from historical literature. The main task is structured as multiple-choice questions targeting moral inference, with carefully crafted distractors that challenge models to go beyond shallow, extractive question answering. To further stress-test model robustness, we introduce adversarial variants designed to surface LLM vulnerabilities and shortcuts due to issues such as data contamination. Our findings show that, while larger models outperform smaller ones, they remain susceptible to adversarial manipulation and often rely on superficial patterns rather than true moral reasoning. This brittleness results in significant self-contradiction, with the best models refuting their own answers in roughly 20% of cases depending on the framing of the moral choice. Interestingly, reasoning-enhanced models fail to bridge this gap, suggesting that scale - not reasoning ability - is the primary driver of performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
