FOLIO: Natural Language Reasoning with First-Order Logic
Yale University（耶鲁大学）；Harvard University（哈佛大学）；NVIDIA（英伟达）；Iowa City West High School（爱荷华城西部高中）；University of Washington（华盛顿大学）；University of Hong Kong（香港大学）；Penn State University（宾夕法尼亚州立大学）；Salesforce Research（Salesforce研究院）；Meta AI
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Sol… 展开作者
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Solano, Hannah Szabo, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev
Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premises and conclusions is ensured by their FOL annotations, which are automatically verified by an FOL inference engine. In addition to the main NL reasoning task, NL-FOL pairs in FOLIO constitute a new NL-FOL translation dataset. Our experiments on FOLIO systematically evaluate the FOL reasoning ability of supervised fine-tuning on medium-sized language models. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models. Our results show that a subset of FOLIO presents a challenge for one of the most capable {Large Language Model (LLM)} publicly available, GPT-4.
Enhancing Logical Reasoning in Language Models via Symbolically-Guided Monte Carlo Process Supervision
School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦大学玛丽女王学院电子工程与计算机科学学院）；The Alan Turing Institute（艾伦·图灵研究所）
Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework
National University of Singapore（国立新加坡大学）；University of Auckland（奥克兰大学）；University of Arizona（亚利桑那大学）；University of California, Santa Barbara（加州大学圣芭芭拉分校）；MBZUAI
LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision
National University of Singapore（新加坡国立大学）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）；University of Melbourne（墨尔本大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）
DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy
XiaoMi AI Lab（小米AI实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Electronic Science and Technology of China（电子科技大学）
SALA: Semantic-Aware Logical Alignment for Complex Reasoning in In-Context Learning
School of Software Engineering, Sun Yat-sen University（中山大学软件工程学院）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；Merchants Union Consumer Finance Company Limited（招商局联融消费金融股份有限公司）
Logical Structure as Knowledge: Enhancing LLM Reasoning via Structured Logical Knowledge Density Estimation
Huzhou University（湖州大学）；Zhejiang Key Laboratory of Intelligent Education Technology and Application（浙江智能教育技术与应用重点实验室）；Banbu AI Foundation（Banbu AI基金会）；Baichuan Inc（白川公司）；University of Edinburgh（爱丁堡大学）；Zhejiang University（浙江大学）
