迷失于逻辑之中：对大型语言模型在LSAT逻辑游戏上的推理能力评估
Lost in the Logic: An Evaluation of Large Language Models' Reasoning Capabilities on LSAT Logic Games
AI总结：
本文通过构建LSAT逻辑游戏数据集并引入Reflexion提示框架，评估并提升了GPT-4和GPT-3.5在复杂逻辑推理任务上的表现，揭示了LLMs修正逻辑错误的能力。
摘要：
在本论文中，我评估了大型语言模型（LLMs）在法学院入学考试（LSAT）上的表现，具体针对该考试的逻辑游戏部分。我聚焦于这一部分，是因为它呈现了一项复杂的逻辑推理任务，因此是评估现代且能力日益增强的LLMs如何处理困难逻辑推理任务的宝贵数据来源。我构建了一个包含LSAT逻辑游戏及其相关元数据的数据集，并在思维链（Chain-of-Thought）提示设置下广泛评估了LLMs的表现。鉴于该设置下表现较弱，我在数据集的一个较小子集上探索了其他提示框架，并将Reflexion的思想适配到该任务中。这在该数据子集上带来了显著提升的准确率：GPT-4达到70%，GPT-3.5达到46%，突显了LLMs尽管初始表现较弱，仍具备修正其逻辑错误的能力。最后，我分析了模型表现更好或更差的逻辑游戏类型，以及通过人工标注观察到的逻辑错误类型，从而为LLMs的逻辑推理能力提供了详细的见解。
英文摘要：
In this thesis, I evaluate the performance of Large Language Models (LLMs) on the Law School Admissions Test (LSAT), specifically the Logic Games section of the test. I focus on this section because it presents a complex logical reasoning task and thus is a valuable source of data for evaluating how modern, increasingly capable LLMs can handle hard logical reasoning tasks. I construct a dataset of LSAT logic games and their associated metadata, and extensively evaluate LLMs' performance in a Chain-of-Thought prompting setting. Given the weak performance in this setting, I explore other prompting frameworks on a smaller subset of the dataset, adapting ideas from Reflexion to this task. This results in a substantially improved accuracy of 70 percent for GPT-4 and 46 percent for GPT-3.5 on this data subset, highlighting the capacity of LLMs to revise their logical errors, despite initially weak performance. Finally, I analyze the types of logic games that models perform better or worse on, as well as the types of logical errors I observe from human annotation, providing detailed insights on the logical reasoning capabilities of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
