本文探讨 GPT-3 在成文法推理任务上的能力，使用 SARA 数据集测试多种提示方法，虽优于先前结果，但在简单合成成文法上表现不佳，揭示其推理局限。
摘要：
成文法推理是根据事实和成文法（由立法机关用自然语言书写的规则）进行推理的任务，是一项基本的法律技能。在本文中，我们探讨了能力最强的 GPT-3 模型 text-davinci-003 在既有成文法推理数据集 SARA 上的表现。我们考虑了多种方法，包括动态少样本提示、思维链提示和零样本提示。虽然我们使用 GPT-3 取得的结果优于先前已发表的最佳结果，但我们也识别出它犯下的几类明显错误。我们调查了这些错误发生的原因。我们发现，GPT-3 对 SARA 所依据的美国实际成文法具有不完善的前置知识。更重要的是，我们创建了简单的合成成文法，确保 GPT-3 在训练期间未曾见过这些成文法。我们发现，GPT-3 在回答有关这些简单合成成文法的直接问题时表现不佳。
英文摘要：
Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we explore the capabilities of the most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider a variety of approaches, including dynamic few-shot prompting, chain-of-thought prompting, and zero-shot prompting. While we achieve results with GPT-3 that are better than the previous best published results, we also identify several types of clear errors it makes. We investigate why these errors happen. We discover that GPT-3 has imperfect prior knowledge of the actual U.S. statutes on which SARA is based. More importantly, we create simple synthetic statutes, which GPT-3 is guaranteed not to have seen during training. We find GPT-3 performs poorly at answering straightforward questions about these simple synthetic statutes.
LegalMALR:Multi-Agent Query Understanding and LLM-Based Reranking for Chinese Statute Retrieval
Faculty of Data Science（数据科学学院）；City University of Macau（澳门城市大学）；Shenzhen Key Laboratory for High Performance Data Mining（深圳高性能数据挖掘重点实验室）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究所）；Chinese Academy of Sciences（中国科学院）；Southern University of Science and Technology（南方科技大学）；Artificial Intelligence Research Institute（人工智能研究院）；Shenzhen University of Advanced Technology（深圳大学先进技术学院）
Foundation Models for Geospatial Reasoning: Assessing Capabilities of Large Language Models in Understanding Geometries and Topological Spatial Relations
GeoDS Lab, Department of Geography, University of Wisconsin-Madison, USA（威斯康星大学麦迪逊分校地理系GeoDS实验室）；Department of Geography and Regional Research, University of Vienna, Austria（维也纳大学地理与区域研究系）；Department of Geography, University of California-Santa Barbara, USA（加州大学圣塔芭芭拉分校地理系）
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
Olin College of Engineering（奥林工程学院）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；University of Oxford（牛津大学）；University College London（伦敦大学学院）；ShanghaiTech University（上海科技大学）；Columbia University（哥伦比亚大学）；University of Manchester（曼彻斯特大学）
Transformer-based Language Models for Reasoning in the Description Logic ALCQ
National and Kapodistrian University of Athens（雅典国立卡波迪斯特里亚大学）；Boston University（波士顿大学）；Athens Technology Center（雅典技术中心）；Archimedes/Athena RC（阿基米德/雅典娜研究中心）
GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations
Department of Informatics, Athens University of Economics and Business（信息学院，雅典经济与商业大学）；Archimedes, Athena Research Center（阿提卡研究中心-阿基米德）；Athena Research Center（阿提卡研究中心）；University of Sheffield（谢菲尔德大学）
JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models
School of Information, Renmin University of China（中国人民大学信息学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；iFLYTEK Research（讯飞研究院）；iFLYTEK AI Research (Central China)（讯飞人工智能研究（华中））
An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
Medical University of Vienna（维也纳医科大学）；Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；KTH Royal Institute of Technology（KTH皇家理工学院）；Humboldt University Berlin（柏林洪堡大学）
Mind meets machine: Unravelling GPT-4's cognitive psychology
Nowrosjee Wadia College（诺罗斯吉·瓦迪亚学院）；Indian Institute of Tropical Meteorology（印度热带气象研究所）；Defence Institute Of Advanced Technology（国防先进技术研究所）；Queen Mary University of London（伦敦玛丽女王大学）
